I’m a bit frustrated with FTP. I have always had a difficult time migrating large amounts of files from one server to another using FTP. As an example, I have a website which has over 60,000 files totaling 6.8 GBs and I’m trying to migrate it to a new server. Both are windows server 2003 with a nice fat connection to the Internet. Every time I’ve tried to ftp it over it breaks at various points. I’ve tried CuteFTP 8 Professional as well as FileZilla.
At this point I’m looking for something I can just set and forget. Something reliable and something that if the connect breaks, it will reconnect and continue remembering where it last left off. I would also like something I can have email me when complete if there is such a program. I’m willing to pay for this product but again, I want it to be reliable. I’m still searching for tools but in the meantime, if anyone knows of such a product and most importantly has experience with using the tool on large sites with no problems, please let me know.
Vince Collins
January 30, 2008 at 12:03 pm
The easiest solution I’ve found is to just zip all the contents up. The programs you mentioned above should have no problems resuming an upload if the connection breaks.
You can also split the archive file into managable chunks to make it easier to move the stuff.
This always seems to simplify the process for me.
January 30, 2008 at 12:41 pm
I’ve done this a few ways - depending on what kind of access you have to each machine.
- Use a removable drive, and then copy back to other server
- If you have backup services (datacenter) have them back it up and do a restore on the other box
- Look at something like DeltaCopy (a rsync clone for Windows)
- Microsoft has the File Server Migration Toolkit - which can be used to migrate data from an older server to 2003, not sure if it would go 2003 to 2003?
- As Dan mentioned - most of the ZIP programs will allow you to define a ’split’ point to break up the data into smaller files…
January 30, 2008 at 2:59 pm
@thecrumb:
“Use a removable drive, and then copy back to other server”
That’s called Broadband SneakerNet.
January 31, 2008 at 9:11 am
Thanks all,
Zipping before FTPing is a decent suggestion, and one I’m about to try. I can’t help to think however that this is a band aid solution.
My servers are in another state and there is no option to attach a portable harddrive to it to copy the content from one server to another. The data center won’t do this unfortunately.
Most of my experience is with Windows servers using Windows FTP over a span of 11 years and it has always been temperamental. Do you think different FTP server software running on Windows could improve this issue? Is FTP with Linux servers considered more stable?
January 31, 2008 at 11:34 am
[...] Vincent Collins’ Weblog Web application development with ColdFusion. Self employed and enjoying it. « FTPing Large Amount of Directories and Files [...]
January 31, 2008 at 12:02 pm
The IIS FTP server sucks. Same folks who make the Filezilla client make a server (free). Might be worth a try.
January 31, 2008 at 12:06 pm
Thanks for that note thecrumb, about IIS http://FTP. Do you use Filezilla’s server yourself? If not, what do you use? My guess is Linux from your response
January 31, 2008 at 12:14 pm
Well, WinRaring the entire site directory and then FTPing the one file worked flawlessly.
I can’t help but think that the various FTP clients might not be designed to manage a 60k file directory structure. I used CuteFTP to transfer the one 5GB file over and never lost connection. Thanks for everyone’s input!
February 1, 2008 at 10:33 am
@Vincent:
Glad it worked. One problem with transferring lots of small files is all the individual FTP connections that are made–they inevitably cause problems.
February 1, 2008 at 10:53 am
Yes, I suppose I shouldn’t blame CuteFTP, or FileZilla, or Windows http://FTP. I’m aware of a few who have written me saying that FTP, regardless of OS is sometimes flaky when dealing with large amounts of files. Zipping into one file did fix the problem so in the end, I win!
Thanks everyone for your help!