MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

How to help yourself when NuGet goes down

Today will be remembered as the day that NuGet.org went down and broke quite some builds. While many people would love to see the NuGet team wearing a pink sombrero, there is something to say about wearing it yourself if you did not manage to workaround this. Let me explain…

First of all, just as with the Azure downtime on a leap day, whenever you rely on an external system and make it mission critical, you should design for failure. You need to anticipate downtime. I’m sure the NuGet team does everything within its power to fix this and is going to inform us whenever they can, but give them some credit please: we’re all human beings making mistakes, that’s how we learn.

pinksombrero

How do I know it’s not just me?

Here’s what you can do

The best thing one can and should do, again, is to anticipate. Have a backup repository, or mirror packages on a MyGet.org feed. Looking at my twitter streams, it is striking to see how many did not think about it.

If you have your packages in source control, impact is limited to not being able to upgrade your packages or install new packages. If you don’t have your packages in your VCS, and you did not anticipate, you might get lucky enough and fix your builds by targeting the local NuGet cache.

Simply register the following path as a NuGet package source and target this one:

%Localappdata%\NuGet\Cache

Avoid spending your day fingers crossed!

Where MyGet is going

Time flies. MyGet has been online for over half a year! We’ve started MyGet as a proof-of-concept which kept growing, thanks to you. You’re one of the more than 600 NuGet feeds we are currently hosting in two datacenters worldwide!

New user interface design, new blog

We’ve “moved your cheese around”. We’ve been redesigning our UI and added a few usability improvements here and there.

Next to that, our new blog went live at blog.myget.org. We will be using it to announce new features, provide sample use cases for MyGet and share other information with you.

More features

It have been a couple of busy weeks. The new release of MyGet which we've just deployed contains an important number of new features. Some small ones, such as the ability to delete all non-latest packages from a feed at once.
We've also added an often requested one: the ability to include packages in your MyGet feed from a different source such as Chocolatey or Orchard Gallery. Xavier blogged about this in MyGet tops Vanilla NuGet feeds with a Chocolatey flavor.

Building on that feature, you can link another feed and include a complete or filtered set of packages from that feed. This offers the ability to have one single package source endpoint which aggregates different external feeds and is enriched with your privately hosted packages. Maarten blogged about the why and how in his post on MyGet package source proxy (beta).

Let us know about your experiences with these new features. You posted these features on our UserVoice feedback forum, we are looking forward to hearing more at UserVoice!

Introducing subscriptions

Since we want to build much more features and improvements, we're looking for some funding to buy ourselves time to do that. We’re introducing subscriptions: a free plan, which still allows you to do what you’ve been doing today. Our small, medium and large plans offer some additional capabilities to your feed. No worries! We’ve promised you to host your feeds and we’ll keep doing that.

Checkout the available plans at www.myget.org/plans and discover which plan we've assigned to you for two months as a gift to say thank you for using MyGet.

It's your product

Know that you can reach out to us through UserVoice or info@myget.org! Let us know your thoughts, feature requests or anything else you would like to share! MyGet is your product, and we want to make sure it stays that way. We’re just the code monkeys making sure you can host your packages and dependencies in a reliable and useful way.

Happy packaging!

Maarten & Xavier
The MyGet Team

Introducing MyGet package source proxy (beta)

My blog already has quite the number of blog posts around MyGet, our NuGet-as-a-Service solution which my colleague Xavier and I are running. There are a lot of reasons to host your own personal NuGet feed (such as protecting your intellectual property or only adding approved packages to the feed, but there’s many more as you can <plug>read in our book</plug>). We’ve added support for another scenario: MyGet now supports proxying remote feeds.

Up until now, MyGet required you to upload your own NuGet packages and to include packages from the NuGet feed. The problem with this is that you either required your team to register multiple NuGet feeds in Visual Studio (which still is a good option) or to register just your MyGet feed and add all packages your team is using to it. Which, again, is also a good option.

With our package source proxy in place, we now provide a third option: MyGet can proxy upstream NuGet feeds. Let’s start with a quick diagram and afterwards walk you through a scenario elaborating on this:

MyGet Feed Proxy Aggregate Feed Connector

You are seeing this correctly: you can now register just your MyGet feed in Visual Studio and we’ll add upstream packages to your feed automatically, optionally filtered as well.

Enabling MyGet package source proxy

Enabling the MyGet package source proxy is very straightforward. Navigate to your feed of choice (or create a new one) and click the Package Sources item. This will present you with a screen similar to this:

MyGet hosted package source

From there, you can add external (or MyGet) feeds to your personal feed and add packages directly from them using the Add package dialog. More on that in Xavier’s blog post. What’s more: with the tick of a checkbox, these external feeds can also be aggregated with your feed in Visual Studio’s search results. Here’s the magical add dialog and the proxy checkbox:

Add package source proxy

As you may see, we also offer the option to filter upstream packages. For example, the filter string substringof('wp7', Tags) eq true that we used will filter all upstream packages where the tags contain “wp7”.

What will Visual Studio display us? Well, just the Windows Phone 7 packages from NuGet, served through our single-endpoint MyGet feed.

Conclusion

Instead of working with a number of NuGet feeds, your development team will just work with one feed that is aggregating packages from both MyGet and other package sources out there (NuGet, Orchard Gallery, Chocolatey, …). This centralizes managing external packages and makes it easier for your team members to find the packages they can use in your projects.

Do let us know what you think of this feature! Our UserVoice is there for you, and in fact, that’s where we got the idea for this feature from in the first place. Your voice is heard!

MyGet tops Vanilla NuGet feeds with a Chocolatey flavor

We recently deployed an all new version of MyGet.org, which contains quite a lot of optimizations and some new features as well. If you didn’t notice, go check it out!

My personal favorite is in fact the underlying architecture that allows us to aggregate feeds and link package sources. These package source presets are configurable on Feed level through the new Package Sources tab available in the Feed management interface.

managePackageSources

To add a package from these package source onto your own feed (either referenced or mirrored, with or without its dependencies), navigate to the Add Package dialog and select the From Feed tab (previously called ‘From NuGet.org’).

addChocolateyPackage

The default setting is still NuGet.org obviously, but you might notice the dropdown containing another feed: Chocolatey.org! That’s right, why not add Chocolatey to your package sources and build a feed containing your favorite tools?

Build your own favorite Chocolatey tools feed

Building such feed is very straightforward. Choose a feed name (which will be represented in your URL) and provide a meaningful description.

myChocolateyFeedDetails

All that is left for a basic MyGet feed is to choose one of the predefined feed templates, as shown below. Obviously you can modify and tweak these settings further to meet your needs afterwards. We’ve updated our FAQ with a full explanation of MyGet’s security model and how you can assign or revoke user rights on a feed.

myChocolateyFeedTemplate

Once the feed is created, you can start pushing packages to it. We support various ways of doing so, including:

  • Uploading packages through the website (and optionally mirror any dependencies found on the configured package source)
  • Referencing or mirroring packages from another feed (any package source, such as NuGet.org, Chocolatey.org, …)
  • Uploading a packages.config file, targeting any package source

Let’s add some packages from the Chocolatey Gallery shall we? Simply select the Chocolatey Gallery package source and type any package name (or any other search criteria using our other search options). Autocomplete will kick in after you typed a character or two and show you a list of possible matches. Select the one you want, verify whether you want to only reference it (copy the metadata onto your feed and keep the real package in the package source) or mirror it (deep copy). If you perform a deep copy, you might prefer to opt-in and check the include dependencies checkbox.

addChocolateyPackageGitExtensions

After clicking the upload button, the operation is queued into the background for processing. It might take up to a minute until the package (and its dependencies if requested) appear on your feed. Note that we also cache the packages list on the website, but nevertheless, once processed they will appear instantly onto your feed when queried from within Visual Studio for instance.

For this demo, I only added GitExtensions to the feed but I mirrored it and included dependencies. After processing, my feed packages list contains the following packages.

myFavoriteToolsPackagesList

Now it’s a piece of cake to add those other tools I’m using. You might be wondering…

Why do you want to do that?

Very simple: I don’t want to mess around with a script or packages.config file on multiple computers. If I repaved my system, that file is not on there. If I want to put it on that repaved machine, I need to find it back (that’s usually the real issue). And if I somehow manage to do so, it’s usually out-of-date.

What if there was a central (read: cloudy) location where I could keep track of this list? A single place to manage the tools list, and always the same location to refer to. Something that would reduce the installation of all my favorite tools to a one-liner. When drafting the Chocolatey chapter in our Pro NuGet book, I learned about the convenient –all command line option support by many of Chocolatey’s commands, including the update (cup) and list (clist) commands . That’s when it struck me that this switch was missing for the install (cinst) command, so I bothered Rob Reynolds (@ferventcoder) with it :) Rob was so nice to help us out and made sure the cinst –all –source [feedUrl] scenario would be supported in the near future. Guess what, it is! Thanks again Rob!

Knowing this, it’s pretty straightforward to repave a system and get all your favorite tools installed in no time. It suffices to run the following command (replace my feed with your feed):

cinst -all -source http://www.myget.org/F/myfavoritetools

Not only can you use it after repaving your system, you could use it as well every time you work for a new customer and need to set up your development environment (maybe have multiple feeds for various scenarios?), or why not share the feed with your team members and make sure everyone benefits from these awesome tools out there.

If ever you have a question about MyGet or need further assistance to get you started, please refer to our blog, reach out on twitter (@MyGetTeam) or use the Support form to contact us. We’ll be happy to help!

Publishing symbol packages for a MyGet feed

MyGet host your NuGet feed serverEver since NuGet 1.2, there is a great way for NuGet package authors to let their users debug into the package’s binaries. With almost no additional effort, package authors can publish their symbols and sources, and package consumers can debug into them from Visual Studio, simply by pushing a symbols package in addition to the standard NuGet package.

SymbolSourceToday, we’re proud to announce MyGet has partnered with SymbolSource.org to offer an easy workflow to publish symbol packages for a private MyGet feed. This means from now on you can publish symbol packages for your private feeds as well!

On a sidenote: we're sharing API keys between both services. If you also want to share the same password with both services, simply go to your MyGet profile page and re-enter your password. We'll keep it in sync after that.

Publishing a symbols package for use with MyGet

As I will assume you are used to publishing packages to NuGet and SymbolSource, here’s what changes. First of all, you will require the URLs to which to publish. Log in to MyGet and browse to your feed details. The Feed Details tab will give you all the information you need, as you can see in the following screenshot:

image

In short, your feed URL remains the same. If you want to consume your private feed in Visual Studio or using the NuGet Package Manager Console, simply add http://www.myget.org/F/yourfeedname as the source. The thing that changed is the publish URL: if you want to publish your packages to MyGet, use the URL http://www.myget.org/F/yourfeedname/api/v1 as the publish URL. For symbol packages, your URL will be in the form of http://nuget.gw.symbolsource.org/MyGet/yourfeedname.

The publish workflow to publish the SamplePackage.1.0.0.nupkg to a MyGet feed, including symbols, would be issuing the following two commands from the console:

1 nuget push SamplePackage.1.0.0.nupkg 00000000-0000-0000-0000-00000000000 -Source http://www.myget.org/F/somefeed/api/v1 2 3 nuget push SamplePackage.1.0.0.Symbols.nupkg 00000000-0000-0000-0000-00000000000 -Source http://nuget.gw.symbolsource.org/MyGet/somefeed

An example of these commands can also be found on the Feed Details tab for your MyGet feed.

Consuming symbol packages in Visual Studio

When logging in to MyGet, you can find the symbols URL compatible with Visual Studio under the Feed Details tab for your MyGet feed. This URL will be the same for all feeds you are allowed to consume, so no need to configure 10+ symbol servers in Visual Studio. Here’s how to configure it.

First of all, Visual Studio typically will only debug your own source code, the source code of the project or projects that are currently opened in Visual Studio. To disable this behavior and to instruct Visual Studio to also try to debug code other than the projects that are currently opened, open the Options dialog (under the menu Tools > Options). Find the Debugging node on the left and click the General node underneath. Turn off the option Enable Just My Code. Also turn on the option Enable source server support. This usually triggers a warning message but it is safe to just click Yes and continue with the settings specified.

MyGet symbol server in Visual Studio

Keep the Options dialog opened and find the Symbols node under the Debugging node on the left. In the dialog shown in Figure 4-14, add the symbol server URL for your MyGet feed: http://srv.symbolsource.org/pdb/MyGet/username/11111111-1111-1111-1111-11111111111. After that, click OK to confirm configuration changes and consume symbols for NuGet packages.

Enjoy!

Setting up a NuGet repository in seconds: MyGet public feeds

A few months ago, my colleague Xavier Decoster and I introduced MyGet as a tool where you can create your own, private NuGet feeds. A couple of weeks later we introduced some options to delegate feed privileges to other MyGet users allowing you to make another MyGet user “co-admin” or “contributor” to a feed. Since then we’ve expanded our view on the NuGet ecosystem and moved MyGet from a solution to create your private feeds to a service that allows you to set up a NuGet feed, whether private or public.

Supporting public feeds allows you to set up a structure similar to www.nuget.org: you can give any user privileges to publish a package to your feed while the user can never manage other packages on your feed. This is great in several scenarios:

  • You run an open source project and want people to contribute modules or plugins to your feed
  • You are a business and you want people to contribute internal packages to your feed whilst prohibiting them from updating or deleting other packages

Setting up a public feed

Setting up a public feed on MyGet is similar to setting up a private feed. In fact, both are identical except for the default privileges assigned to users. Navigate to www.myget.org and sign in using an identity provider of choice. Next, create a feed, for example:

Create a MyGet NuGet feed and host your own NuGet packages

This new feed may be named “public”, however it is private by obscurity: if someone knows the URL to the feed, he/she can consume packages from it. Let’s change that. Go to the “Feed Security” tab and have a look at the assigned privileges for Everyone. By default, these are set to “Can consume this feed”, meaning that everyone can add the feed URL to Visual Studio and consume packages. Other options are “No access” (requires authentication prior to being able to consume the feed) and “Can contribute own packages to this feed”. This last one is what we want:

Setting up a NuGet feed

Assigning the “Can contribute own packages to this feed” privilege to a specific user or to everyone means that the user (or everyone) will be able to contribute packages to the feed, as long as the package id used is not already on the feed and as long as the package id was originally submitted by this user. Exactly the same model as www.nuget.org, that is.

For reference, all available privileges are:

  • Has no access to this feed (speaks for itself)
  • Can consume this feed (allows the user to use the feed in Visual Studio / NuGet)
  • Can contribute own packages to this feed '(allows the user to contribute packages but can only update and remove his own packages and not those of others)
  • Can manage all packages for this feed (allows the user to add packages to the feed via the website and via the NuGet push API)
  • Can manage users and all packages for this feed (extends the above with feed privilege management capabilities)

Contributing to a public feed

Of course, if you have a public feed you may want to have people contributing to it. This is very easy: provide them with a link to your feed editing page (for example, http://www.myget.org/Feed/Edit/public). Users can publish their packages via the MyGet user interface in no time.

If you want to have users push packages using nuget.exe or NuGet Package Explorer, provide them a link to the feed endpoint (for example, http://www.myget.org/F/public/). Using their API key (which can be found in the MyGet profile for the user) they can push packages to the public feed from any API consumer.

Enjoy!

 

PS: We’re working on lots more, but will probably provide that in a MyGet Premium version. Make sure to subscribe to our newsletter on www.myget.org if this is of interest.

NuGet push... to Windows Azure

When looking at how people like to deploy their applications to a cloud environment, a large faction seems to prefer being able to use their source control system as a source for their production deployment. While interesting, I see a lot of problems there: your source code may not run immediately and probably has to be compiled. You don’t want to maintain compiled assemblies in source control, right? Also, maybe some QA process is in place where a deployment can only occur after approval. Why not use source control for what it’s there for: source control? And how about using a NuGet repository as the source for our deployment? Meet the Windows Azure NuGetRole.

Disclaimer/Warning: this is demo material and should probably not be used for real-life deployments without making it bullet proof!

Download the sample code: NuGetRole.zip (262.22 kb)

How to use it

If you compile the source code (download), you have X steps left in getting your NuGetRole running on Windows Azure:

  • Specifying the package source to use
  • Add some packages to the package source feed (which you can easily host on MyGet)
  • Deploy to Windows Azure

When all these steps have been taken care of, the NuGetRole will download all latest package versions from the package source specified in ServiceConfiguration.cscfg:

1 <?xml version="1.0" encoding="utf-8"?> 2 <ServiceConfiguration serviceName="NuGetRole.Azure" 3 xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" 4 osFamily="1" 5 osVersion="*"> 6 <Role name="NuGetRole.Web"> 7 <Instances count="1" /> 8 <ConfigurationSettings> 9 <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" /> 10 <Setting name="PackageSource" value="http://www.myget.org/F/nugetrole/" /> 11 </ConfigurationSettings> 12 </Role> 13 </ServiceConfiguration>

Packages you publish should only contain a content and/or lib folder. Other package contents will currently be ignored by the NuGetRole. If you want to add some web content like a default page to your role, simply publish the following package:

NuGet Package Explorer MyGet NuGet NuGetRole Azure

Just push, and watch your Windows Azure web role farm update their contents. Or have your build server push a NuGet package containing your application and have your server farm update itself. Whatever pleases you.

How it works

What I did was create a fairly empty Windows Azure project (download).  In this project, one Web role exists. This web role consists of nothing but a Web.config file and a WebRole.cs class which looks like the following:

1 public class WebRole : RoleEntryPoint 2 { 3 private bool _isSynchronizing; 4 private PackageSynchronizer _packageSynchronizer = null; 5 6 public override bool OnStart() 7 { 8 var localPath = Path.Combine(Environment.GetEnvironmentVariable("RdRoleRoot") + "\\approot"); 9 10 _packageSynchronizer = new PackageSynchronizer( 11 new Uri(RoleEnvironment.GetConfigurationSettingValue("PackageSource")), localPath); 12 13 _packageSynchronizer.SynchronizationStarted += sender => _isSynchronizing = true; 14 _packageSynchronizer.SynchronizationCompleted += sender => _isSynchronizing = false; 15 16 RoleEnvironment.StatusCheck += (sender, args) => 17 { 18 if (_isSynchronizing) 19 { 20 args.SetBusy(); 21 } 22 }; 23 24 return base.OnStart(); 25 } 26 27 public override void Run() 28 { 29 _packageSynchronizer.SynchronizeForever(TimeSpan.FromSeconds(30)); 30 31 base.Run(); 32 } 33 }

The above code is essentially wiring some configuration values like the local web root and the NuGet package source to use to a second class in this project: the PackageSynchronizer. This class simply checks the specified NuGet package source every few minutes, checks for the latest package versions and if required, updates content and bin files.  Each synchronization run does the following:

1 public void SynchronizeOnce() 2 { 3 var packages = _packageRepository.GetPackages() 4 .Where(p => p.IsLatestVersion == true).ToList(); 5 6 var touchedFiles = new List<string>(); 7 8 // Deploy new content 9 foreach (var package in packages) 10 { 11 var packageHash = package.GetHash(); 12 var packageFiles = package.GetFiles(); 13 foreach (var packageFile in packageFiles) 14 { 15 // Keep filename 16 var packageFileName = packageFile.Path.Replace("content\\", "").Replace("lib\\", "bin\\"); 17 18 // Mark file as touched 19 touchedFiles.Add(packageFileName); 20 21 // Do not overwrite content that has not been updated 22 if (!_packageFileHash.ContainsKey(packageFileName) || _packageFileHash[packageFileName] != packageHash) 23 { 24 _packageFileHash[packageFileName] = packageHash; 25 26 Deploy(packageFile.GetStream(), packageFileName); 27 } 28 } 29 30 // Remove obsolete content 31 var obsoleteFiles = _packageFileHash.Keys.Except(touchedFiles).ToList(); 32 foreach (var obsoletePath in obsoleteFiles) 33 { 34 _packageFileHash.Remove(obsoletePath); 35 Undeploy(obsoletePath); 36 } 37 } 38 }

Or in human language:

  • The specified NuGet package source is checked for packages
  • Every package marked “IsLatest” is being downloaded and deployed onto the machine
  • Files that have not been used in the current synchronization step are deleted

This is probably not a bullet-proof solution, but I wanted to show you how easy it is to use NuGet not only as a package manager inside Visual Studio, but also from your code: NuGet is not just a package manager but in essence a package management protocol. Which you can easily extend.

One thing to note: I also made the Windows Azure load balancer ignore the role that’s updating itself. This means a roie instance that is synchronizing its contents will never be available in the load balancing pool so no traffic is sent to the role instance during an update.

Why MyGet uses Windows Azure

MyGet - NuGet hosting private feedRecently one of the Tweeps following me started fooling around and hit one of my sweet spots: Windows Azure. Basically, he mocked me for using Windows Azure for MyGet, a website with enough users but not enough to justify the “scalability” aspect he thought Windows Azure was offering. Since Windows Azure is much, much more than scalability alone, I decided to do a quick writeup about the various reasons on why we use Windows Azure for MyGet. And those are not scalability.

First of all, here’s a high-level overview of our deployment, which may illustrate some of the aspects below:

image

Costs

Windows Azure is cheap. Cheap as in cost-effective, not as in, well, sleezy. Many will disagree with me but the cost perspective of Windows Azure can be real cheap in some cases as well as very expensive in other cases. For example, if someone asks me if they should move to Windows Azure and they now have one server running 300 small sites, I’d probably tell them not to move as it will be a tough price comparison.

With MyGet we run 2 Windows Azure instances in 2 datacenters across the globe (one in the US and one in the EU). For $180.00 per month this means 2 great machines at two very distant regions of the globe. You can probably find those with other hosters as well, but will they manage your machines? Patch and update them? Probably not, for that amount. In our scenario, Windows Azure is cheap.

Feel free to look at the cost calculator tool to estimate usage costs.

Traffic Manager

Traffic Manager, a great (beta) product in the Windows Azure offering allows us to do geographically distributed applications. For example, US users of MyGet will end up in the US datacenter, European users will end up in the EU datacenter. This is great, and we can easily add extra locations to this policy and have, for example, a third location in Asia.

Next to geographically distributing MyGet, Traffic Manager also ensures that if one datacenter goes down, the DNS pool will consist of only “live” datacenters and thus provide datacenter fail-over. Not ideal as the web application will be served faster from a server that’s closer to the end user, but the application will not go down.

One problem we have with this is storage. We use Windows Azure storage (blobs, tables and queues) as those only cost $0.12 per GB. Distributing the application does mean that our US datacenter server has to access storage in the EU datacenter which of course adds some latency. We try to reduce this using extensive caching on all sides, but it’d be nicer if Traffic Manager allowed us to setup georeplication for storage as well. This only affects storing package metadata and packages. Reading packages is not affected by this because we’re using the Windows Azure CDN for that.

CDN

The Windows Azure Content Delivery Network allows us to serve users fast. The main use case for MyGet is accessing and downloading packages. Ok, the updating has some latency due to the restrictions mentioned above, but if you download a package from MyGet it will always come from a CDN node near the end user to ensure low latency and fast access. Given the CDN is just a checkbox on the management pages means integrating with CDN is a breeze. The only thing we’ve struggled with is finding an acceptable caching policy to ensure stale data is limited.

Windows Azure AppFabric Access Control

MyGet is not one application. MyGet is three applications: our development environment, staging and production. In fact, we even plan for tenants so every tenant in fact is its own application. To streamline, manage and maintain a clear overview of which user can authenticate to which application via which identity provider, we use ACS to facilitate MyGet authentication.

To give you an example: our dev environment allows logging in via OpenID on a development machine. Production allows for OpenID on a live environment. In staging, we only use Windows Live ID and Facebook whereas our production website uses different identity providers. Tenants will, in the future, be given the option to authenticate to their own ADFS server, we’re pretty sure ACS will allow us to simply configure that and instrument only tenant X can use that ADFS server.

ACs has been a great time saver and is definitely something we want to use in future project. It really eases common authentication pains and acts as a service bus between users, identity providers and our applications.

Windows Azure AppFabric Caching

Currently we don’t use Windows Azure AppFabric Caching in our application. We currently use the ASP.NET in-memory cache on all machines but do feel the need for having a distributed caching solution. While appealing, we think about deploying Memcached in our application because of the cost structure involved. But we might as well end up with Wndows Azure AppFabric Caching anyway as it integrates nicely with our current codebase.

Conclusion

In short, Windows Azure is much more than hosting and scalability. It’s the building blocks available such as Traffic Manager, CDN and Access Control Service that make our lives easier. The pricing structure is not always that transparent but if you dig a little into it you’ll find affordable solutions that are really easy to use because you don’t have to roll your own.

Continuous Package Integration: NuGet vs Source Control

Update (August 17, 2011):

David Fowler created an awesome NuGetPowerTools package that streamlines this process further. Also check out David Ebbo's post for more info.

One of the questions I receive regularly when talking about enterprise approaches for using NuGet, is the following one:

"Why don't you put the NuGet packages in source control as well?"

In my opinion a very valid question to ask, and I reached the point where I realized it's better to write a blogpost on it once and use it for future reference :-)

Please note: what I will discuss here is my take on it, and not the holy grail!

Storing NuGet packages in source control?

As David Ebbo already explained, there are approaches to not store the NuGet packages in source control. Your question is: why?

One could say that disk space is cheap! That's a valid approach if you have no issues repeating your dependencies in source control every single time. I'd like to see however how that works out for you if you have let's say 500 line of business apps in your SCM, all having the same dependency to your favorite logging library of around 0.5Mb. That's 250Mb worth of storage! This also slows down the time needed to perform a backup, or the disk space required to store your backups, thus increasing your costs in storage and power consumption. Now extrapolate this reasoning to all your dependencies..

Add on top of this that some SCM's don't manage binary files that well (TFS anyone?): ask your devs how much time they lost already fighting with a system that is unable to do binary diffing? You ever experienced upgrading an assembly in TFS from version X to version Y? Forgot to explicitly check out the file, replaced it on disk, to find out the system tells you it found no changes to commit? Don't get me wrong, TFS is a great tool, but its source control system could be better.

For some, this is a reason not to use TFS, but I'd say that any Source Control system is meant to store sources, not binaries. Use a document management system if you need document versioning and take benefit from its search features for instance. Assemblies explicitly tell you their AssemblyVersion, so why would you want history on those files?

What you really want to is to have history on your dependencies! You want to be able to say: version A of my app depends on version X of a dependency, version B of my app depends on version Y of that same dependency, we upgraded X to Y on that date when person P was working on feature F... That's the interesting information, not the actual binary.

Now people argue: Yeah, but I want my builds to be reproducable, a given changeset (pardon my wording: revision) should always produce the same output. Agreed! But does that mean that the dependencies over which you don't have any control need to be in source control? You know, those libraries that already are versioned, the information already compiled and baked into the binary file? Nope, just reference them, you'll only find one binary that matches that specific version of the library out there (I'm coming back on this point later in this post). Now that's exactly the kind of information you'll find in the repositories.config and packages.config files that NuGet uses to keep track of your package dependencies.

A benefit of storing the packages in source control is that its consumers don't need to rely on NuGet. This is one of the main principles behind NuGet, as outlined by Phill Haack in the announcement of NuGet. That is nice indeed, and it is cool that you can do that, but should you? There are cases where you should, but I find them rare. The main reason for me to store those packages in source control would be because you don't want to wait for that choice to be made :-) Team A kicking off a new project could start using it, while Team B has no time because it's dealing with featuritis. In such situations you could benefit from storing the packages to avoid impacting other teams. 

In my opinion, using NuGet or supporting teams that want to use it, should preferably be a company-wide strategic choice. For the same reason, you probably won't mix IDEs (Visual Studio, SharpDevelop, Notepad, VIM, other? :-)) amongst your team members, or mix source control systems (TFS, Git, SVN, ...), build servers, ...

So here is my statement:

  • don't store the binaries in source control: keep NuGet packages out
  • only store the metadata in source control: keep track of repositories.config and packages.config files
  • have a strategy: standardising the way you embrace the power of NuGet inside the organisation will benefit you all and avoid headaches

Continuous (Package) Integration

Now, as stated earlier: every build of a given changeset/revision of your project should always produce the same output. How does this work when you depend on files which are not in source control?

Given the fact you do store what you depend on (metadata), all you need to do is fetch those dependencies in a pre-build step. This is what I call Continuous Package Integration.

Note: I'm speaking about CI builds here by example. I use the exact same approach for any other type of build: QA builds, Release builds, ...

This setup implies you have a corporate NuGet server running, preferably even with multiple feeds. There are multiple ways to accomplish this, but if you want a quick start and play with this setup, I suggest you try out MyGet (NuGet-as-a-Service) and focus on your process first. MyGet can also help in ensuring a specific package version is always available by mirroring packages from the official NuGet feed.

Optimization

Now, it might seem cumbersome to continuously fetch those packages every single time you do a build, but what prevents you of setting up a caching mechanism on your build agents? Now its my turn to tell you disk space is cheap :-) (with the addition that it's used as a local cache: no need for backup).

This could be very simply accomplished with a few PowerShell scripts hooking into the build steps, that check the local cache first, and if not present fetch it. This mechanism will be self-organizing and become faster as we use it (don't you like that?).

The same counts for your development machines: have a script and local cache for the packages you use. Run a pre-build that checks if any repositories.config or packages.config file has changed and updates your local environment.

Please, feel free to comment and share your thoughts on this matter!