Boston Linux & UNIX was originally founded in 1994 as part of The Boston Computer Society. We meet on the third Wednesday of each month at the Massachusetts Institute of Technology, in Building E51.

BLU Discuss list archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss] large log file transfer

Why split it?  It depends on how big big is.  Even compressed by gzip -9 if
the file is 'too large', splitting it in half (or more parts) to get the
size of each compressed portion below the threshold of data transfer 'pain'
can make it possible to transfer large files over 'limited' links or with
'limited size media'.  I first ran into split back when using 128K floppy
diskettes to move files.  The concept is still the same, just the
definition of 'large' has changed.

On Tue, Jun 23, 2015 at 10:17 AM, Mike Small <smallm at> wrote:

> John Malloy <jomalloy at> writes:
> > I have a log file (325MB)  that I need to transfer from a restricted
> > network, that I cannot plug a USB into.
> >
> > Is there an easy way to "split up" the log file into smaller chunks and
> Zip
> > it to get it over the net?
> So Jerry and Jack have pointed you to split, but why do you want to
> split it into chunks?
> I must admit I've not dealt with any networks lately that I couldn't
> reliably download a 300 MB file on, but is it not still possible to
> resume interrupted transfers?  Are you using vanilla ftp, http or scp?
> If you're using scp, I'd be inclined not to zip it but to use ssh's
> compression so when it finally does arrive I wouldn't have to unzip and
> then again find something else to do while that happens. But I guess scp
> doesn't support resume (hmmm, maybe ssh dd and adjust the offset based
> on the file size after the failure? -- perhaps that's more trouble than
> splitting).
> By "restricted network" do you mean that rsync is not an option?
> --
> Mike Small
> smallm at
> _______________________________________________
> Discuss mailing list
> Discuss at

><> ... Jack

The Four Boxes of Liberty - "There are four boxes to be used in the defense
of liberty: soap, ballot, jury and ammo. Please use in that order."
"Whatever you do, work at it with all your heart"... Colossians 3:23
"Anyone who has never made a mistake, has never tried anything new." -
Albert Einstein
"You don't manage people; you manage things. You lead people." - Admiral
Grace Hopper, USN
"Tell me and I forget. Teach me and I remember. Involve me and I learn." -
Ben Franklin

BLU is a member of BostonUserGroups
BLU is a member of BostonUserGroups
We also thank MIT for the use of their facilities.

Valid HTML 4.01! Valid CSS!

Boston Linux & Unix /