MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Continuous Package Integration: NuGet vs Source Control

Update (August 17, 2011):

David Fowler created an awesome NuGetPowerTools package that streamlines this process further. Also check out David Ebbo's post for more info.

One of the questions I receive regularly when talking about enterprise approaches for using NuGet, is the following one:

"Why don't you put the NuGet packages in source control as well?"

In my opinion a very valid question to ask, and I reached the point where I realized it's better to write a blogpost on it once and use it for future reference :-)

Please note: what I will discuss here is my take on it, and not the holy grail!

Storing NuGet packages in source control?

As David Ebbo already explained, there are approaches to not store the NuGet packages in source control. Your question is: why?

One could say that disk space is cheap! That's a valid approach if you have no issues repeating your dependencies in source control every single time. I'd like to see however how that works out for you if you have let's say 500 line of business apps in your SCM, all having the same dependency to your favorite logging library of around 0.5Mb. That's 250Mb worth of storage! This also slows down the time needed to perform a backup, or the disk space required to store your backups, thus increasing your costs in storage and power consumption. Now extrapolate this reasoning to all your dependencies..

Add on top of this that some SCM's don't manage binary files that well (TFS anyone?): ask your devs how much time they lost already fighting with a system that is unable to do binary diffing? You ever experienced upgrading an assembly in TFS from version X to version Y? Forgot to explicitly check out the file, replaced it on disk, to find out the system tells you it found no changes to commit? Don't get me wrong, TFS is a great tool, but its source control system could be better.

For some, this is a reason not to use TFS, but I'd say that any Source Control system is meant to store sources, not binaries. Use a document management system if you need document versioning and take benefit from its search features for instance. Assemblies explicitly tell you their AssemblyVersion, so why would you want history on those files?

What you really want to is to have history on your dependencies! You want to be able to say: version A of my app depends on version X of a dependency, version B of my app depends on version Y of that same dependency, we upgraded X to Y on that date when person P was working on feature F... That's the interesting information, not the actual binary.

Now people argue: Yeah, but I want my builds to be reproducable, a given changeset (pardon my wording: revision) should always produce the same output. Agreed! But does that mean that the dependencies over which you don't have any control need to be in source control? You know, those libraries that already are versioned, the information already compiled and baked into the binary file? Nope, just reference them, you'll only find one binary that matches that specific version of the library out there (I'm coming back on this point later in this post). Now that's exactly the kind of information you'll find in the repositories.config and packages.config files that NuGet uses to keep track of your package dependencies.

A benefit of storing the packages in source control is that its consumers don't need to rely on NuGet. This is one of the main principles behind NuGet, as outlined by Phill Haack in the announcement of NuGet. That is nice indeed, and it is cool that you can do that, but should you? There are cases where you should, but I find them rare. The main reason for me to store those packages in source control would be because you don't want to wait for that choice to be made :-) Team A kicking off a new project could start using it, while Team B has no time because it's dealing with featuritis. In such situations you could benefit from storing the packages to avoid impacting other teams. 

In my opinion, using NuGet or supporting teams that want to use it, should preferably be a company-wide strategic choice. For the same reason, you probably won't mix IDEs (Visual Studio, SharpDevelop, Notepad, VIM, other? :-)) amongst your team members, or mix source control systems (TFS, Git, SVN, ...), build servers, ...

So here is my statement:

  • don't store the binaries in source control: keep NuGet packages out
  • only store the metadata in source control: keep track of repositories.config and packages.config files
  • have a strategy: standardising the way you embrace the power of NuGet inside the organisation will benefit you all and avoid headaches

Continuous (Package) Integration

Now, as stated earlier: every build of a given changeset/revision of your project should always produce the same output. How does this work when you depend on files which are not in source control?

Given the fact you do store what you depend on (metadata), all you need to do is fetch those dependencies in a pre-build step. This is what I call Continuous Package Integration.

Note: I'm speaking about CI builds here by example. I use the exact same approach for any other type of build: QA builds, Release builds, ...

This setup implies you have a corporate NuGet server running, preferably even with multiple feeds. There are multiple ways to accomplish this, but if you want a quick start and play with this setup, I suggest you try out MyGet (NuGet-as-a-Service) and focus on your process first. MyGet can also help in ensuring a specific package version is always available by mirroring packages from the official NuGet feed.

Optimization

Now, it might seem cumbersome to continuously fetch those packages every single time you do a build, but what prevents you of setting up a caching mechanism on your build agents? Now its my turn to tell you disk space is cheap :-) (with the addition that it's used as a local cache: no need for backup).

This could be very simply accomplished with a few PowerShell scripts hooking into the build steps, that check the local cache first, and if not present fetch it. This mechanism will be self-organizing and become faster as we use it (don't you like that?).

The same counts for your development machines: have a script and local cache for the packages you use. Run a pre-build that checks if any repositories.config or packages.config file has changed and updates your local environment.

Please, feel free to comment and share your thoughts on this matter!

Copy packages from one NuGet feed to another

Copy packages from one NuGet feed to another - MyGet NuGet Server

Yesterday, a funny discussion was going on at the NuGet Discussion Forum on CodePlex. Funny, you say? Well yes. Funny because it was about a feature we envisioned as being a must-have feature for the NuGet ecosystem: copying packages from the NuGet feed to another feed. And funny because we already have that feature present in MyGet. You may wonder why anyone wants to do that? Allow me to explain.

Scenarios where copying packages makes sense

The first scenario is feed stability. Imagine you are building a project and expect to always reference a NuGet package from the official feed. That’s OK as long as you have that package present in the NuGet feed, but what happens if someone removes it or updates it without respecting proper versioning? This should not happen, but it can be an unpleasant surprise if it happens. Copying the package to another feed provides stability: the specific package version is available on that other feed and will never change unless you update or remove it. It puts you in control, not the package owner.

A second scenario: enhanced speed! It’s still much faster to pull packages from a local feed or a feed that’s geographically distributed, like the one MyGet offers (US and Europe at the moment). This is not to bash any carriers or network providers, it’s just physics: electrons don’t travel that fast and it’s better to have them coming from a closer location.

But… how to do it? Client side

There are some solutions to this problem/feature. The first one is a hard one: write a script that just pulls packages from the official feed. You’ll find a suggestion on how to do that here. This thing however does not pull along dependencies and forces you to do ugly, user-unfriendly things. Let’s go for beauty :-)

Rob Reynolds (aka @ferventcoder) added some extension sauce to the NuGet.exe:

NuGet.exe Install /ExcludeVersion /OutputDir %LocalAppData%\NuGet\Commands AddConsoleExtension NuGet.exe addextension nuget.copy.extension NuGet.exe copy castle.windsor –destination http://myget.org/F/somefeed

Sweet! And Rob also shared how he created this extension (warning: interesting read!)

But… how to do it? Server side

The easiest solution is to just use MyGet! We have a nifty feature in there named “Mirror packages”. It copies the selected package to your private feed, distributes it across our CDN nodes for a fast download and it pulls along all dependencies.

Mirror a NuGet package - Copy a NuGet package

Enjoy making NuGet a component of your enterprise workflow! And MyGet of course as well!

Delegate feed privileges to other users on MyGet

MyGetOne of the first features we had envisioned for MyGet and which seemed increasingly popular was the ability to provide other users a means of managing packages on another user’s feed.

As of today, we’re proud to announce the following new features:

  • Delegating feed privileges to other users – This allows you to make another MyGet user “co-admin” or “contributor” to a feed. This eases management of a private feed as that work can be spread across multiple people.
  • Making private feeds private by requiring authentication – It’s now possible to configure a feed so that nobody can consult its list of packages unless a valid login is provided. This feature is not yet available for use with NuGet 1.4.
  • Global deployment – We’ve updated our deployment so managing feeds can now be done on a server that’s closer to you.

Now when is Microsoft going to buy us out :-)

Delegating feed privileges to other users

MyGet now allows you to make another MyGet user “co-admin” or “contributor” to a feed. This eases management of a private feed as that work can be spread across multiple people. If combined with the “private feeds” option, it’s also possible to give some users read access to the feed while unauthenticated users can not access the feed created.

To delegate privileges to a user, navigate to the feed details and click the Feed security tab. This tab allows you to change feed privileges for different users. Adding feed privileges can be done by clicking the Add feed privileges… button (duh!).

Add MyGet feed privileges

Available privileges are:

  • Has no access to this feed (speaks for itself)
  • Can consume this feed (allows the user to use the feed in Visual Studio / NuGet)
  • Can manage packages for this feed (allows the user to add packages to the feed via the website and via the NuGet push API)
  • Can manage users and packages for this feed (extends the above with feed privilege management capabilities)

After selecting the privileges, the user receives an e-mail in which he/she can claim the acquired privileges:

Claim MyGet feed privileges

Privileges are not granted per direct: after assigning privileges, the user has to claim these privileges by clicking a link in an automated e-mail that has been sent.

Making private feeds private by requiring authentication

It’s now possible to configure a feed so that nobody can consult its list of packages unless a valid login is provided. Combined with the feed privilege delegation feature one can granularly control who can and who can not consume a feed from MyGet. Note that his feature is not yet available for use with NuGet 1.4, we hope to see support for this shipping with NuGet 1.5.

In order to enable this feature, on the Feed security tab change feed privileges for Everyone to Has no access to this feed.

NuGet feed authentication

This will instruct MyGet to request for basic authentication when someone accesses a MyGet feed. For example, try our sample feed: http://www.myget.org/F/mygetsample/

Global deployment

We’ve updated our deployment so managing feeds can now be done on a server that’s closer to you. Currently we have a deployment running in a European datacenter and one in the US. We hope to expand this further as well as leverage a content delivery network for high-speed distribution of packages.

image

We need your opinion!

As features keep popping into our head, the time we have to work on MyGet in our spare time is not enough. To support some extra development, we are thinking along the lines of introducing a premium version which you can host in your own datacenter or on a dedicated cloud environment. We would love some feedback on the following survey:

MyGet now compatible with NuGet Package Explorer v1.5

Most of you will agree that Package Explorer is a major part within the NuGet ecosystem. In preparation for the latest version 1.5 release, Luan Nguyen (aka dotNetJunky) pointed us to an incompatibility issue with MyGet (thanks again for that!).

A new package property IsLatestVersion was added and Package Explorer depends on it for the improved Select package dialog as explained here.

I'm glad to announce that MyGet is now using this property as well, with a slight switch to it.

In the NuGet Gallery, a package with IsLatestVersion=true simply means: it is the latest one... duh! :-) for the record: the latest official one!

For MyGet, we use this property within the scope of the specific MyGet feed: this means, it is the latest version available within that specific feed. Actually, this is exactly the same behavior as within the NuGet Gallery, but the meaning is different because it concerns a private feed on MyGet. The MyGet feed is (currently) unaware of new versions that might get published on the official one. So unless you upload or add a newer package to your MyGet feed, the latest version within your private feed might get out-of-sync with the latest version within the NuGet Gallery.

Let me illustrate this with an example: let's say SomeAwesomePackage shipped a first version 1.0 in the NuGet Gallery which you added to your MyGet feed. This will be flagged with IsLatestVersion = true (it's the only version too). You later add a newer version 1.1 of the package, which in turn will be flagged as the latest version, and will unflag the 1.0 version. Now, your happy with this version and didn't upgrade for a couple of months, while version 1.2 has been made available in the NuGet Gallery. At this point in time, your latest version in the MyGet feed (v 1.1) is not the latest version from the NuGet Gallery (v 1.2). Just trying to point out the difference here :-)

To query your feed, just copy/paste your MyGet feed url into the dialog as shown below.

The little checkbox at the bottom saying "Only show latest version of each package Id." is interacting with the IsLatestVersion property of the packages.

Enjoy!

Adding NuGet packages from the official feed to your MyGet feed: some improvements

One of the things we want to improve on MyGet is the add-package functionality from the official NuGet feed. We felt this user experience could be better, so here's a first step!

First of all, the default search behavior has changed (and we hope improved as well!):

  • the term you enter in the search box is used now to scan the NuGet package ID and Title only
  • the default search method is StartsWith (self-explanatory I hope?)
  • uppercasing or lowercasing doesn't matter (we do a ToLower behind the scenes anyway)
  • by default, we now only search through the latest versions
You'll notice there are a bit more options in the UI as well, so you can adjust the behavior to your needs.

Some of the search settings are now optional:

  • search through the package Summary field
  • search through the package Description field
  • search through all versions of all packages
The moment you type at least two characters, an autocomplete box will display with your matching results, as shown below:
In a second phase, I hope to add some more useful functionality to this feature, such as search by Author, OSS license type, ...
Feel free to suggest the ones you feel are really missing.

Get your local NuGet repository online in a private MyGet feed

One of the MyGet features I've been working on lately should make it easier to populate your private feed with the NuGet packages you want.

One of the specific use cases for MyGet is to be able to quickly set up a private feed with your own packages, which you might have in a local repository on your computer.

Before MyGet, you could make this feed available to other computers by sharing your repository for instance on a network drive, or by hosting your own NuGet server.

This required you to do some plumbing to get a server running, or to manage folder security settings, and above all, if a computer trying to consume the feed was not in the network, it could not get any packages from you repository.

MyGet provides you with a service (NuGet-as-a-Service) that allows you to easily host your private NuGet feeds and have it always accessible from any computer connected to the internet, without that setup hassle.

Since we launched, you were already able to upload a single package at a time to your feed. Maarten extended this with a simple checkbox to define whether or not you want to include its dependencies. Note that this dependency resolution only works for dependencies to packages on the official NuGet feed at the moment.

To facilitate the upload progress, I've now extended it to allow you to upload multiple packages at once.

multi-package-upload

This saves you again a couple of clicks and redirects!

When the packages are successfully uploaded to your feed, you'll get a nice notification telling you which packages have been added.

multi-package-upload success

Noticed that the above upload of 2 packages results in 3 packages being added? (it resolved a dependency to elmah.corelibrary 1.2 for elmah 1.2.0.1)

MyGet now supports pushing from the command line

One of the work items we had opened for MyGet was the ability to push packages to a private feed from the command line. Only a few hours after our initial launch, David Fowler provided us with example code on how to implement NuGet command line pushes on the server side. An evening of coding later, I quickly hacked this into MyGet, which means that we now support pushing packages from the command line!

For those that did not catch up with my blog post overload of the past week: MyGet offers you the possibility to create your own, private, filtered NuGet feed for use in the Visual Studio Package Manager.  It can contain packages from the official NuGet feed as well as your private packages, hosted on MyGet. Want a sample? Add this feed to your Visual Studio package manager: http://www.myget.org/F/chucknorris

Pushing a package from the command line to MyGet

The first thing you’ll be needing is an API key for your private feed. This can be obtained through the “Edit Feed” link, where you’ll see an API key listed as well as a button to regenerate the API key, just in case someone steals it from you while giving a demo of MyGet :-)

image

Once you have the API key, it can be stored into the NuGet executable’s settings by running the following command, including your private API key and your private feed URL:

1 NuGet setApiKey c18673a2-7b57-4207-8b29-7bb57c04f070 -Source http://www.myget.org/F/testfeed

After that, you can easily push a package to your private feed. The package will automatically show up on the website and your private feed. Do note that this can take a few minutes to propagate.

1 NuGet push RouteMagic.0.2.2.2.nupkg -Source http://www.myget.org/F/testfeed

More on the command line can be found on the NuGet documentation wiki.

Other change: authentication to the website

Someone on Twitter (@corydeppen) complained he had to login using Windows Live ID. Because we’re using the Windows Azure AppFabric Access Control Service (which I’ll abbreviate to ACS next time), this was in fact a no-brainer. We now support Windows Live ID, Google, Yahoo! and Facebook as authentication mechanisms for MyGet. Enjoy!

Creating your own private NuGet feed: MyGet

myget - NuGet as a serverEver since NuGet came out, I’ve been thinking about leveraging it in a corporate environment. I've seen two NuGet server implementations appear on the Internet: the official NuGet gallery server and Phil Haack’s NuGet.Server package. As these both are good, there’s one thing wrong with them: you can't be lazy! You have to do some stuff you don’t always want to do, namely: configure and deploy.

After discussing some ideas with my colleague Xavier Decoster, we decided it’s time to turn our heads into the cloud: we’re providing you NuGet-as-a-Service (NaaS)! Say hello to MyGet.

MyGet offers you the possibility to create your own, private, filtered NuGet feed for use in the Visual Studio Package Manager.
It can contain packages from the official NuGet feed as well as your private packages, hosted on MyGet. Want a sample? Add this feed to your Visual Studio package manager: http://www.myget.org/F/chucknorris

We've already received some feature requests, but feel free to send us your own most-wanted features or report bugs. 

Chuck Norris Feed

Feel free to go ahead and create your private feed. Some ideas for possible scenarios (more at Xavier's site):

  • A feed containing only the packages you or your company often use
  • A feed containing only your (open-source?) project and its dependencies
  • A feed containing just a few packages that you want to use for a certain project: tell your developers to just install them all

Bugs and feature requests? Feel free to post them as a comment below. 

Enjoy http://myget.org!