If you are interested in the theory behind the mag I can highly recommend the book The Magnetics Measurement Handbook by J.M. Janicke. Besides a good coverage of magnetics theory, it contains much of what you need to build a useful fluxgate magnetometer, plus other devices. Note - Jack Janicke passed away and his book is no longer being published.

Besides the fluxgate mag, the proton precession mag is also a do-able home project. An early article by Nicholas Wadsworth has been the basis for several projects such as Phil Barnes' PPM. Dan Fountain has also been working on one, and sooner or later I will build one to compare with my fluxgate unit. There is a good list server discussion forum you should join if you want to see what other people are doing.