LISTSERV mailing list manager LISTSERV 15.5

Help for SOCNET Archives


SOCNET Archives

SOCNET Archives


View:

Next Message | Previous Message
Next in Topic | Previous in Topic
Next by Same Author | Previous by Same Author
Chronologically | Most Recent First
Proportional Font | Monospaced Font

Options:

Join or Leave SOCNET
Reply | Post New Message
Search Archives


Subject: Re: size limits in R?
From: "Mark S. Handcock" <[log in to unmask]>
Reply-To:Mark S. Handcock
Date:Tue, 8 Jan 2008 22:53:55 -0800
Content-Type:text/plain
Parts/Attachments:
Parts/Attachments

text/plain (117 lines)


*****  To join INSNA, visit http://www.insna.org  *****

Hi Uri,

R, and the 'network' package in particular, are pretty good with memory. For
example the code:

library(network)
a <- network.initialize(n=14000)
b <- matrix(sample(1:network.size(a),size=2.1*70000,replace=T),ncol=2)
index <- b[,1]+100000*b[,2]
b <- b[!duplicated(index) & b[,1] != b[,2],][1:70000,]
sim70k <- add.edges(a,b[,1],b[,2])
summary(sim70k)

produces a network with 14k nodes and 70k edges that takes up about 36Mb.
However if a full sociomatrix is naively stored it takes about 14000*13999*4
= 694 Mbytes (so 'network' uses only 5% of this storage). An edge list alone
could take about 70000*2*4= 0.5Mb. In sum, 'network' does a good job storing
networks, giving it stores for computation purposes.

However some functions in R may be written to use the full sociomatrix form,
which means it does not take to many sociomatrices to use up the RAM on a
computer. I think the current plotting functions are like that as they are
meant for visualization of smaller networks (you need good eyes to
distinguish 70K nodes on letter paper).

So R need not be bad on memory. 'network' is pretty good, and some functions
have yet to take advantage of the strengths of the 'network' class. Kudos to
Carter for 'network' - It is a real accomplishment.

Mark

-------------------------------------------------
Mark S. Handcock
Professor and Chair of Statistics
Department of Statistics, B-313 Padelford Hall
University of Washington, Box 354322     Phone:  (206) 543-6774
Seattle, WA  98195-4322.                 FAX:    (360) 365-6324
Web:  www.stat.washington.edu/~handcock
internet:  [log in to unmask]


On 1/8/08 3:58 PM, "Robert Ackland" <[log in to unmask]> wrote:

> *****  To join INSNA, visit http://www.insna.org  *****
> 
> Hi Uri,
> 
> I investigated R (the sna/network packages by Carter Butts, in particular)
> for large graphs and ran into the problem you describe.  I'm pretty sure
> that Carter mentions somewhere in the documentation that there are
> efficiency issues with large graphs.  Perhaps there are other R packages
> that handle large graphs, but for the time being I've been using the boost
> graph library (http://www.boost.org/libs/graph/) which is definitely able
> to handle large graphs but of course then one needs to work in
> C++ and hand-code SNA routines, which is probably not what most people
> want to do.
> 
> Rob
> 
> -------------------------------------
> Dr Robert Ackland
> Fellow, Australian Demographic and Social Research Institute
> College of Arts and Social Sciences
> The Australian National University
> 
> e-mail:       [log in to unmask]
> homepage:     http://adsri.anu.edu.au/people/robert.php
> project site: http://voson.anu.edu.au
> ph./fax/mob.: +61 2 6125 0312/+61 2 6125 2992/+61 438 833 525
> mail:         Coombs Building, 9
>               Canberra, ACT 0200
>               AUSTRALIA
> -------------------------------------
> 
> On Tue, 8 Jan 2008, Uri Shwed wrote:
> 
>> *****  To join INSNA, visit http://www.insna.org  *****
>> 
>> Hi All,
>> How does one deals with a really large network (14k nodes, 70k edges)?
>> I'm trying my first steps in R - I thought it could do anything, but
>> this size scares it away "Error: protect(): protection stack overflow"
>> for edgelist and "Error: cannot allocate vector of size 1523603 Kb" for
>> plot request. I have raised R's max memory to the point it tells me not
>> to be silly...Should my relationship with R be terminated at this early
>> stage?
>> I'd love to compose a succinct summary of replies and repost for the
>> public good.
>> Thanks,
>> 
>> -- 
>> Uri Shwed
>> ISERP, Columbia University
>> Room 805, 420 W 118th St.
>> New York 10027 NY
>> Tel: 212 854 9374
>> 
>> _____________________________________________________________________
>> SOCNET is a service of INSNA, the professional association for social
>> network researchers (http://www.insna.org). To unsubscribe, send
>> an email message to [log in to unmask] containing the line
>> UNSUBSCRIBE SOCNET in the body of the message.
>> 
> 
> _____________________________________________________________________
> SOCNET is a service of INSNA, the professional association for social
> network researchers (http://www.insna.org). To unsubscribe, send
> an email message to [log in to unmask] containing the line
> UNSUBSCRIBE SOCNET in the body of the message.

_____________________________________________________________________
SOCNET is a service of INSNA, the professional association for social
network researchers (http://www.insna.org). To unsubscribe, send
an email message to [log in to unmask] containing the line
UNSUBSCRIBE SOCNET in the body of the message.

Back to: Top of Message | Previous Page | Main SOCNET Page

Permalink



WWW.LISTS.UFL.EDU

CataList Email List Search Powered by the LISTSERV Email List Manager