One great advantage to running Ghost 2003 or 8.3 (2005) from a DOS boot disk is that you can also do some pretty massive cleanups on your HD before making the Ghost backup.

Once you've booted up your system into DOS, you can run a program called "NTFS4DOS" which will allow you to see the files and folders on your NTFS HD from the DOS prompt.
I've written for myself, a few little batch files to remove crapola from the HD before backing it up.  I've added those batch files to my Ghost boot disk.

What files, you may ask.  Well there's the Pagefile.sys which can be 1.5x the size of your installed RAM.  On my PC that's three gigs.  Then there's all the old Restore Points, that can also take up a huge amount of space, and of course all the temp files, etc. that Windows stacks up on your HD.
By cleaning up all this junk, you can get your Ghost backup into much less space than if you did nothing.  I'm still getting my own Ghost backup onto a single DVD, or 4 gig Flash Drive.

So many options are available to anyone wanting to be a bit inventive.

Ghost rocks!

Happy New Years Everyone!
Shadow 

First, you don't need to clean out the pagefile or hibernation files for Ghost's benefit, in any case. The data in those files never goes into an image (the file name will appear in it, but the data is always left out and takes no space because Ghost knows it's not important to keep).

Quote:

Yes, although it's quite complicated to work through all the details. Here's an incredibly simplified sketch of what's involved in modern OS's..

Every process involves data of several kinds; a lot of it is able to be shared between processes, like the executable code in DLLs. That stuff is read-only, and is already sitting around in files already, and so doesn't involve the pagefile. The other main kind is private data, unique to that one specific process, and that does involve the pagefile.

The first thing to understand is that the pagefile initially set up for you by the OS is done the way it is to reserve the space. It can grow later if needed, so the initial allocation isn't critical. However, because the pagefile sees an awful lot of traffic it's one of the worst things in the system to have fragmented. And extending the pagefile is extremely slow, and while it's being done the program that triggered the growth can't run at all, which is bad if it's time-critical. Hence, the beefy initial allocation to stop that situation potentially happening.

The second thing is that programs themselves tend to work similarly; they ask the OS to reserve room for them to grow into later, since when programs start they don't know what they will be asked to do in future while they run. Since asking for more memory is slow, and since programs would have to handle being told they can't have more (in which case they may have to outright fail) modern programs end up asking the OS for a large amount of potential room at startup so they can avoid asking later.

[ ]

The end result is that when a program asks for some "memory", what the OS actually does is just mark out a section of the pagefile for it to potentially use in future. The pagefile has to be big enough to cope with this "headline" figure of all the memory every program wants to reserve for future use, just in case.

Now, within those reservations, at any point in time there's the amount of memory that the programs have actually done something with thus far, as opposed to merely having reserved for the future. Windows calls this committed memory, and the total amount of committed memory in the system is the "commit charge".

And within that number, there's the amount that any one program is using right now, or at least within some recent time, as opposed to data that it's done something with but not recently. This number is called the working set, and this is the number that is displayed as "Mem Usage" in the "Processes" tab of the Task Manager in Windows XP, and the operating system is dynamically recomputing it all the time based on what you do.

Data in the working set has to be live in the actual physical memory so that the CPU can work on it; so, that number represents the contribution from programs to the load of physical memory. The programs are all competing not only with each other, but also with the part of the OS that looks after the disks and caches file data for that memory, and the OS is trying to balance the constantly changing demands.

The working set is usually much, much smaller than the commit charge. By evicting data that's not in the working set of one process, the OS makes room for another process or for itself to use to accelerate file access. Making room for caching the filesystem is hugely important, and that's why you normally want a lot more physical memory around than the working set load, and thus why the working set computation matters.

The end result is this:
- in theory, if you know what specific programs you run and how many, you can size the pagefile's reserve capacity to suit that particular workload (or indeed, elect to run without one, although that's generally a really bad idea because it blocks the OS using physical memory for disk cache, so it hurts performance). If you're doing that you're also sizing your physical memory carefully too!
- in reality, almost no-one knows what they want to run or what those programs need to use, and so virtually everyone leaves it up to the OS to estimate something.
- the OS default estimate is based on how much physical memory is installed on the basis that if you have more memory, you intend to run more programs with it, and so the pagefile is proportional to the installed physical memory size.

Basically, for most people, it's best to just leave the OS alone to do its thing. Don't even think about it.

Quote:

There isn't one. Really, there isn't.

For very specific applications, for specific purposes, it's possible to end up with very specific needs that are worth tuning. Embedded uses of computers tend to be like that, since they are typically running a small range of programs with known characteristics, what you do in that case is measure the load and then tune based on those measurements.

Otherwise, for home computers put to unknowable workloads, it's really just a matter of guesswork. The default guess made by the OS is just one that they know isn't a bad guess for most people (since if it was, it'd become obvious across hundreds of millions of machines that they did get it wrong). So, unless you have a very specific reason to, it's best to leave that alone.

TheShadow wrote on Dec 29^th, 2007 at 2:47pm:

You've been around a long time then

The first "commercial" computer credited with actually having demand paging was the Ferranti Atlas, circa 1961, and the idea was around in research in the late 50's.

Check out the Atlas Computer Laboratory, built to house the machines and the staff to run and program them. That site is a treasure trove of history and well worth poking around in, and the photos from the 60's are ... well, priceless.

Quote:

The DisablePagingExecutive one? Actually, what that really does is more complicated than that, and it has more to do with drivers than anything else. The kernel code itself is almost completely locked in memory regardless. Still, as long as you have enough memory it's relatively harmless, unlike disabling paging completely.

Implementing demand paging in an operating system is tricky because if you are, for instance, in an interrupt service routine and a page fault occurs you can't go to the disk to get the missing page (because the machine isn't in a state to do anything until you complete the interrupt handler). That's bluescreen territory, game over. In fact, many Windows bluescreens are from buggy drivers whose developers used the above tweak when developing them (because it makes debugging them easier).

Quote:

While that's mostly true for the main file data, there are other kinds of invisible fragmentation in NTFS (and it's truly invisible, no tool reports on it) that Ghost 2003 doesn't entirely completely resolve.

As I noted, fragmentation is a universal property of putting things into containers. And the data about files in NTFS goes into containers (and containers of containers...) too, and that means this can get fragmented too. Indeed, the data that says where the other data is can have this happen (one of the effects of files being fragmented is you need more data to describe where all the pieces ended up instead of a simple "start at this cluster, X clusters long" run description, so this is a perniciously self-reinforcing problem).

When restoring, Ghost has to extensively rewrite all this metadata to ensure it stays consistent and describes the new disk layout. Now, by defragmenting the data, that means that potentially the file meta-data that describes where the other file data is can end up having a completely different structure.

Ghost 2003 is limited as to what it can do in the way of structural rewrites on the metadata, especially if a really large amount of fragmentation of the attributes has occurred. It tries hard, but in some cases what you'll end up with is all the data together on disk (which solves the externally visible *symptoms* of fragmentation) but the meta-data will be more complicated than it strictly needs to be.

[ Incidentally, the 25002/25003 type of internal errors in Ghost result from nasty corner cases of this same kind of fragmentation. If the state of the filesystem before imaging was bad enough, Ghost's meta-data rewrites end up in some chicken-and-egg messes. ]

So, if you defragment regularly, all will stay well. But once the filesystem on a disk passes a certain extreme level of fragmentation (which is hard to quantify exactly) it's sometimes beyond Ghost to undo it all.

Combine that with the fact that Ghost doesn't have access to some of the kind of statistical information an on-line tool can use to optimize placement (as tools like Diskeeper do), there's definitely plenty of room for specialized defragmenters in the toolbox.

zmdmw52 wrote on Dec 29^th, 2007 at 8:03pm:

Correct, from the page file. The page file is the primary store for process-private storage; executables are the primary store for things that are shared.

Your physical memory is primarily a cache for those things; every page in physical memory comes from either a regular file, or is attached to the pagefile.

Quote:

Correct.

The importance of this is that when a program does something which causes a page to be read in from disk, room needs to be made for it and something else gets discarded; not written out, discarded.

Part of the process of working set computation is "cleaning" pages in memory; if they have been written to, meaning that they are different from what's currently on disk, they have to be written to disk to keep the disk and memory in sync. After the write, the page stays valid and can still be used by the program and brought back into the working set without needing to hit the disk - this is called a "soft" page fault.

Quote:

Indeed. Just remember, if you truncate the page file to some number and the demand ever exceeds that the page file will be grown, and it will probably be fragmented, and that's bad.

Quote:

I think you'll find that generally not to be the case. Having it on a separate partition will generally cause more head movement to seek between the pagefile containing process-private space and the executables containing the shared data. That's bad.

Quote:

That does work, because traffic to and from the pagefile for private data and traffic coming in from executables for shared data won't be contending with each other over the position of the disk heads.

And it's even better if the two are on disk channels that can run separately. That's why in days of yore, it was common for some systems to have a dedicated, high-performance SCSI disk just for paging, just as it was for database servers to have such a dedicated disk for the database recovery log.

Quote:

Heh. Like most serious developers of software for Windows I use Process Explorer almost daily, and have for years along with the other SysInternals tools.

The other one you mentioned, well - most of what it claims does not represent the way operating systems actually manage memory. It may or may not actually do something useful, but quite frankly I wouldn't touch it or any one of the dozens of other utilities that make similar claims.

Nbree,
I bow to your Ghost expertise.....
just one detail that maybe I failed to put into this thread (I've said it so many times before, I loose track)  is that I never use NTFS.  Some of your responses referred to that format.

I'm too much of a control freak to ever trust my data to an NTFS partition.
The day I upgraded my OS from Windows 98/SE to XP-pro, I made the decision to keep my FAT-32 format and I've never regretted that decision  for even one second.

Yes, I've been at this stuff way too long.  Growing up with vacc. tube processors in the 60's, transistors in the 70's and finally the PC and LSI in the 80's.  The first computer I ever got my hands on filled two floors of a five story building and heated the entire five story building. 

I built my first PC, an Intel 8088 with 64K of ram.
MS DOS 2.0 was the best I had available then.

Well, that was then and this is now.  
I've been using Ghost since 1997.  The little "Ghost Now" icon in my sig line should be familiar to you.  

I just did my HD cleanup and Ghost backup and restore earlier this evening.
If there's a downside to using Ghost Restore to do my defragging for me, I've never seen any, in all the hundreds of times I've done it over the years.

Ghost rocks!

Happy New Year Everyone!

Shadow  

TheShadow wrote on Dec 28^th, 2007 at 9:47am:

Shadow,

TheShadow wrote on Jul 4^th, 2008 at 9:32am: