MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

NuGet package restore from a secured feed

One of the most frequently asked questions at MyGet is the following one (we have pending updates to our FAQ section):

How do I set up NuGet package restore against a private MyGet feed requiring authentication?

This is also one of the things you might end up doing when debugging NuGet package restore issues.

For public feeds, you only need to change the repository URL in the nuget.targets file to let your build server know from where it needs to fetch the packages. For private feeds however, there are a few things you need to know.

Which credentials should I use?

At MyGet, we recommend you to create a separate account for your build agents and give it specific permission on your feed (e.g. readonly or read/write, but no additional permissions).

It is not a technical requirement though: you could simply use your personal account, but please be aware that in this case you share your credentials!

As you'll see in this post, you can store the credentials for the build service account on the build agent(s) without having to share them with anyone. Using a user's account for the build agent can break anyone's build if access for this user is revoked...

Visual Studio will prompt for credentials

As soon as you try to communicate with a secured package source in Visual Studio, it will prompt you for credentials. So why do you get the following build error when using package restore?

There's no non-interactive way to provide credential parameters

NuGet package restore relies on the NuGet.exe commandline tool by using the install command. The commandline will either prompt you for credentials (which isn't suitable for automated build scenarios), or will look for credentials in nuget.config file in %AppData%\NuGet\nuget.config (if you use the Non-Interactive option).

The latter looks like what you need in automated build scenarios, but requires you to store feed credentials on the machine, for the user account that will perform the build. This can become cumbersome if you have a multitude of solutions using this feature.

Hierarchical NuGet.config doesn't take credentials into account (yet!)

The latest version of NuGet has support for hierarchical nuget.config files, which is an attempt to overcome the need to store everything on the machine. It allows you to have a solution-level NuGet configuration which should be taken into account during package restore.

This means that feed URL and credentials could be stored next to your solution instead of being pre-configured in the user profile. However, credentials aren't picked up (yet), and there's no easy way to store them (encrypted) into any nuget.config file other than the one in your roaming user profile (explained in the next section of this post).

This is a known issue which seems to be fixed in vNext of NuGet. Check this Codeplex issue for more details. Not sure though whether this will be facilitated without having to copy-paste those encrypted credentials from one config to another.

You can store feed credentials in your user-profile NuGet.config

That's likely to be the easiest approach: as you register the package source URL, you might want to save the required credentials as well. This is however not exposed in the Visual Studio NuGet Package Manager extension, so you'll have to use the NuGet.exe commandline tool. The following gist illustrates a few of these options that should help you configure your secured feed, including credentials.

MyGet logo stickers

We're fairly sure the mailman didn't know he was carrying more than one package this morning when handing over the enveloppe. To celebrate our new logo, we've been handing out free subscriptions to our Starter plan during the past few days. Follow us on Twitter for future announcements (our Features page is growing!) and perhaps a chance to win.

Today we're happy to announce we'll be handing out some of these cool stickers during our upcoming trips! Thank you StickerMule for a job well done!

Want to get some? Then don't miss the opportunity to meet our team members and learn how we built MyGet at these upcoming events:

See you there & Happy Packaging!

Update project templates to the latest NuGet packages

We noticed a question on StackOverflow that proved we weren't the only ones finding it a little sub-optimal having to update NuGet packages right after creating a new project.

Most of us are likely to use the default project templates that come with Visual Studio or an SDK. Let's take the example of the MVC4 project template for C#, using Razor syntax.

MVC4 C# Web application template using Razor syntax

 

This project template is consuming quite a few NuGet packages by default. jQuery is one of them. The whole point is that these NuGet packages can be updated more frequently and independent from any pending SDK update or other product release. This is a good thing!

As a direct consequence, this also means that the default templates become "outdated". Outdated is a strong word, as the template itself isn't really outdated, but rather the packages list it wants to consume from NuGet. jQuery is one of those packages that gets very frequent updates. There's an easy way to update all packages in a solution all at once. Use the Package Manager Console, type

Update-Package

and hit ENTER. Done!

But why not avoid this step (or at least partially) and change the defaults?

Note: We'd recommend you to create your own project template so you can always revert back to the default one in case you, or someone else, is going to mess things up :)

All Visual Studio (2012) project templates can be found here:

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ProjectTemplates\

If you want to create some custom project templates, I'd recommend you to create them here:

%USERPROFILE%\Documents\Visual Studio 2012\Templates\ProjectTemplates

In this post, we'll show you how you can change the defaults for the MVC4 CSHTML project template. You can find it here:

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ProjectTemplates\CSharp\Web\1033\MvcWebApplicationProjectTemplatev4.0.cshtml\

To modify the files, you'll have to edit them as Administrator (you know the drill, right-click Notepad++ or Sumblime and click Run As Administrator).

The file you'll want to take a look at is the .vstemplate file. It's an XML file containing template instructions for Visual Studio. Look for a section called packages. It should look something like this:

<WizardData> 
<packages repository="registry" keyName="AspNetMvc4VS11" isPreunzipped="true"> 
<package id="EntityFramework" version="5.0.0" skipAssemblyReferences="true" /> 
<package id="jQuery" version="1.7.1.1" />
...

Let's take jQuery as an example again: we want to upgrade the dependency to version 1.8.2 by default.

To do so, you modify the above snippet to look like this:

<WizardData>
<packages repository="registry" keyName="AspNetMvc4VS11" isPreunzipped="true">
<package id="EntityFramework" version="5.0.0" skipAssemblyReferences="true" />
<package id="jQuery" version="1.8.2" />
...

Easy huh?

Now you found the candy, you can change the default installed package versions, or even add or remove the packages you want. Whatever you do, make sure you don't break the template so proceed with caution. If you remove a package dependency, make sure you remove any dependent configuration or references in the project template's files. If you update a package to a newer version, make sure those dependent configurations are updated as well.

Take a look at the project template's Scripts folder. You see that little _references.js file?

This is a harmless example of things that can be left behind and out-of-sync with the package edits you make. Open the file (run as administrator) and update those references accordingly. The jQuery reference should now be the following:

/// <reference path="jquery-1.8.2.js" />

Ever wondered why you didn't have to be online to be able to create a new MVC project and consume all those packages? Then check this folder and be amazed:

C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET MVC 4\Packages

Obviously, the packages you want to support in your default project templates should be available there as well, so download those NuGet packages and extract them here. You can download the NuGet package after logging in to NuGet.org: look for the package you want, select the version you need, and you'll notice a download link on the left side.

Download a NuGet packages from the Gallery

Once downloaded, unblock the package (right-click, properties, unblock), copy it to the packages directory (C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET MVC 4\Packages). Next, unzip it, and remove all garbage. The relevant content is selected on the following screenshot. Ensure you rename the .nuspec file by adding the version in front of it, e.g. jquery.1.8.2.nuspec.

Relevant package contents

From now on, all newly created default MVC4 CSHTML Web projects will already contain the updated jQuery dependency.

How we push GoogleAnalyticsTracker to NuGet

If you check Maarten’s blog post Tracking API usage with Google Analytics, you’ll see that a small open-source component evolved from MyGet. This component, GoogleAnalyticsTracker, lives on GitHub and NuGet and has since evolved into something that supports Windows Phone and Windows RT as well. Here’s his guest post:


It’s funny how things evolve. GoogleAnalyticsTracker started as a small component inside MyGet, and since a couple of weeks it uses MyGet to publish itself to NuGet. Say what? In this blog post, I’ll elaborate a bit on the development tools used on this tiny component.

Source code

Source code for GoogleAnalyticsTracker can be found on GitHub. This is the main entry point to all activity around this “project”. If you have a nice addition, feel free to fork it and send me a pull request.

Staging NuGet packages

Whenever I update the source code, I want to automatically build it and publish NuGet packages for it. Not directly to NuGet: I want to keep the regular version, the WinRT and WP version more or less in sync regarding version numbers. Also, I sometimes miss something which I fix again 5 minutes after. In the meanwhile, I like to have the generated package on some sort of “staging” feed, at MyGet. It’s even public, check http://www.myget.org/F/githubmaarten if you want to use my development artifacts.

When I decide it’s time for these packages to move to the “official NuGet package repository” at NuGet.org, I simply click the “push” button in my MyGet feed. Yes, that’s a manual step but I wanted to have some “gate” in the middle where I should explicitly do something. Here’s what happens after clicking “push”:

Push to NuGet

That’s right! You can use MyGet as a staging feed and from there push your packages onwards to any other feed out there. MyGet takes care of the uploading.

Building the package

There’s one thing which I forgot… How do I build these packages? Well… I don’t. I let MyGet Build Services.do the heavy lifting. On your feed, you can simply click the “Add GitHub project” button and a list of all your GitHub repos will be shown:

Build GitHub project

Tick a box and you’re ready to roll. And if you look carefully, you’ll see a “Build hook URL” being shown:

MyGet build hook

Back on GitHub, there’s this concept of “service hooks”, basically small utilities that you can fire whenever a new commit occurs on your repository. Wouldn’t it be awesome to trigger package creation on MyGet whenever I check in code to GitHub? Guess what…

GitHub build hook

That’s right! And MyGet even runs unit tests. Some sort of a continuous integration where I have the choice to promote packages to NuGet whenever I think they are stable.

MyGet Build Services - Public Beta

We’re happy to announce that MyGet Build Services is in Public Beta since, well, now. MyGet Build Services enable you to add packages to your feed by just giving us your Git, Mercurial or Subversion repo. We build it, we package it, we publish it. And starting today, every MyGet user can use it.

More information on MyGet Build Services can be found in a previous blog post, although some things have been updated. You’ve provided us feedback through our UserVoice, resulting in the following changes:

  • We now support .NET Framework 4.5. New features available in .NET Framework 4.5 are described at http://msdn.microsoft.com/en-us/library/vstudio/ms171868.aspx.
  • Build hooks! The list of your build definitions now shows a build hook URL which you can link to your GitHub or BitBucket repository. Calling this build hook (HTTP POST) will trigger a new build. Or in other words: a commit will trigger a build.
  • Our build server now checks for a file called build.bat first, then MyGet.sln, then your .sln and then .csproj/.vbproj files (UV)
  • We now support Git, Mercurial and Subversion repositories (UV)
  • You’ll now have some tools available in the environment variables (UV)
  • There’s more information in the build logs (UV)

Keep the feedback coming! Build Services are now enabled on every MyGet account. Do note MyGet Build Services are still a beta, and as Willy Wonka would say… “Little surprises around every corner, but nothing dangerous.”

Happy packaging!

PS: Why not deploy your freshly built apps using Octopus Deploy?

MyGet Build Services - Join the private beta!

Good news! Over the past 4 weeks we’ve been sending out tweets about our secret project MyGet project “wonka”. Today is the day Wonka shows his great stuff to the world… In short: MyGet Build Services enable you to add packages to your feed by just giving us your GitHub repo. We build it, we package it, we publish it.

Our build server searches for a file called MyGet.sln and builds that. No problem if it's not there: we'll try and build other projects then. We'll run unit tests (NUnit, XUnit, MSTest and some more) and fail when those fail. We'll search for packages generated by your solution and if none are generated, we take a wild guess and create them for you.

To make it more visual, here are some screenshots. First, you have to add a build source, for example a GitHub repository (in fact, GitHub is all we currently support):

MyGet Add build source

After that, you simply click “Build”. A couple of seconds or minutes later, your fresh package is available on your feed:

MyGet build package

MyGet package result

If you want to see what happened, the build log is available for review as well:

MyGet build log

Enroll now!

Starting today, you can enroll for our private beta. You’ll get on a waiting list and as we improve build capacity, you will be granted access to the beta. If you’re in, tell us how it behaves. What works, what doesn’t, what would you like to see improved. Enroll for this private beta now via http://www.myget.org/buildservices. Limited seats!

Do note it’s still a beta, and as Willy Wonka would say… “Little surprises around every corner, but nothing dangerous.”

Happy packaging!

PS: Why not deploy your freshly built apps using Octopus Deploy?

Feature highlight: MyGet Gallery

It's been less than two months since we introduced the public MyGet Gallery, and we are very happy to have some awesome projects onboard, often shipping development builds or using MyGet as a staging environment before pushing packages upstream. These feeds deserve more attention and people should know these feeds exist.

Did you know you can get the latest development builds of SignalR and MVC Contrib delivered through MyGet?

If you think your project's development builds deserve a place in our Gallery, you can easily opt-in using your feed's Gallery Settings.

Gallery feed readme

Our latest v1.3 release now allows feed owners to add a feed readme for publication in the MyGet Gallery. This way, feed owners now can easily add some instructions or disclaimers to the gallery feed details page using the well-known markdown format. Don't forget to add a link to your project page and/or sources as well.

You might recognize the pagedown markdown editor there, which is also used on the StackExchange Web sites, so you are probably already familiar with it. In return for this excellent control, we also added the StackOverflow identity provider to our login page. People who prefer to login using their StackOverflow account can now do so, or you can link it to your other identities within your MyGet account as well.

Layout improvements

While at it and based on the invaluable feedback of our users, we also improved the look-n-feel of the packages and feed listings by removing clutter and putting focus on content.

You can now not only see which packages are listed, and what version they're at, but you can also track when this version has become available. A simple click on the icon next to the package will trigger a package download in your browser, whilst the icons on the top right bring you to the SymbolSource symbols repository of this feed (if available) or give access to the RSS feed associated with this package repository.

What's next?

The screenshot below (taken from our latest development builds) is giving you a glimpse at what's coming next: user profiles and activity streams! That's right: you'll be able to get insights into the activity of a public feed or track the history of a given package. Coming soon...

We hope you like it! Keep those suggestions and feedback coming! Who knows, maybe one day you'll see your wish granted :)

Happy packaging!

MyGet Update: version 1.3.0

We are happy to announce a new version of the MyGet Web site containing a set of improvements, fixes and updates.

The following work items have been addressed in this release:

Product

  • number of plans has been reduced. We now have Free, Starter (formerly Small plan), Professional (formerly Large plan) and a new Enterprise plan
  • introduction of the new Enterprise plan focused on companies that require more performance, integration and security

Usability

  • improved layout and readability on smaller screen resolutions
  • auto-lower cased routing for feed URL (helps avoiding issues caused by incorrect casing of feed identifier)
Performance
  • upgraded to MVC4
Features
  • new identity provider: StackExchange
  • upgraded SymbolSource integration to take benefit from their improved and freshly released API
  • Gallery feeds now support a README in Markdown format, available in the feed's Gallery Settings
  • Enterprise plan: upgraded administrative & quota management dashboard
More details about the new features will be covered in follow-up blog posts soon.
 
Happy packaging!

Site issues on July 2nd, 2012

What a day! July 2nd seemed to go from sunshine to rain in a couple of hours. The good news is: we’re back. Let’s first go through what happened…

What happened?

imageThis morning, you've probably received an e-mail stating that "Your Free subscription at MyGet has expired". Free? Expired? Even we were puzzled! We’ve quickly sent out an e-mail stating free subscriptions don’t expire and that we were investigating the issue. In our rush to find out what happened, we even forgot to remove some template text from that e-mail. Professional of us, really.

A couple of hours later, the website started acting funny: one moment it was up, one moment it appeared down. The strange thing was: we did not see this happening in our server monitoring page. The brand new Windows Azure portal was all shiny and bright, all systems running. A short glitch? Probably.

And then 4:00 PM (GMT+1) came around: we started to receive SMS messages from Pingdom. UP, DOWN, UP, DOWN. Some sweaty hours later, we’ve found the root cause of all this mess and shortly thereafter, deployed a hotfix to the cloud. What a day!

All-in-all, we’ve spammed you in the morning, had some small glitches during the day and went bezerk between 4:00 PM (GMT+1) and 7:00 PM (GMT+1).

What was wrong?

During the entire day, we were suspecting two of our latest changes. First of all, we’ve moved storage accounts in Windows Azure to make use of some of the new features in there with regards to mirroring packages. The code isn’t done yet, but we did already switch the storage accounts. Next to that, we’ve recently deployed our site using ASP.NET MVC 4 (RC). Was any of this the cause? We’ve been searching… But no.

Elmah revealed some interesting details. We were getting throttled on those new storage accounts. Was this due to the fact that we enabled storage analytics? We’ve disabled them and found that we were back up in full glory! For a moment, because this wasn’t the reason we were being throttled…

We process several operations asynchronously, such as updating our caches. It seemed that there were 16.000 commands in there, all telling the system to update the cache for one specific feed. One of our newest MyGet members managed to break a couple of gears there! Something we don’t judge. In contrary: there’s no better stress test or user test than real users.

A rookie mistake

Now why were there 16.000 messages unprocessed and why were some of them on the dead-letter queue? And why were our machines looking fine (all green!) while the website did not respond? The answer lies in the following piece of code:

1 [DebuggerDisplay("{Title} Version={Version}")] 2 public class FeedPackage 3 : TableServiceEntity 4 { 5 public string Id { get; set; } 6 public string Version { get; set; } 7 8 // ... 9 10 public override bool Equals(object obj) 11 { 12 if (ReferenceEquals(null, obj)) return false; 13 if (ReferenceEquals(this, obj)) return true; 14 if (obj.GetType() != typeof(FeedPackage)) return false; 15 return GetHashCode() == obj.GetHashCode(); 16 } 17 18 public override int GetHashCode() 19 { 20 int hashCode = 7; 21 22 if (PartitionKey != null) hashCode ^= PartitionKey.GetHashCode(); 23 if (RowKey != null) hashCode ^= RowKey.GetHashCode(); 24 25 return hashCode; 26 } 27 }

Did you spot it? We’ve gone with a complete rookie implementation of Equals and GetHashCode there…

The devil is in the fact that we’re using the hashcode of package A and compare it with package B. That’s a serious problem, as those hash codes are based on the hashcodes of their package identifier and package version. Guess what MSDN tells us

If two string objects are equal, the GetHashCode method returns identical values. However, there is not a unique hash code value for each unique string value. Different strings can return the same hash code.

Our newest member was in the position where he had a lot of packages uploaded to MyGet and by coincidence, two packages appeared to be “equal” using our faulty code above.

Now why did the website choke on that?

Windows Azure table storage, which we use to store all your data, In the .NET space, the WIndows Azure storage client uses WCF Data Service to fetch data. Deep in that framework, the Equals'() method is called on every entity to check whether a specific entity is already being tracked by the data context in use. And since we had a duplicate result for Equals(), an unhandled exception was thrown there.

Wait a minute: you guys don’t catch unhandled exceptions? Yes we do. And when such exception occurs, we retry some operations for up to five times. Retry operations? Well, rather than failing immediately, a good pattern is to retry a failing operation to check that the error wasn’t just a minor glitch like a network fault. On Windows Azure, it is recommended to do this for calls to any external system, like a webserver calling storage. Check David Aiken’s blog for some info on this.

So, retries… Yes. Retry an I/O operation with minor latency for five times. Have a user who’s experiencing an issue with the website F5 a couple of times. And have that queue with 16.000 operations pending hammer this storage account again. What will happen? Throttling. Windows Azure tells our system to retry again after a second. And our retry logic does exactly that. This retry logic causes some threads to sleep, all the queue processing and F5-ing causes some more thread to sleep and what happens? The server comes to a halt while it still looks okay to our external monitoring.

Are we running just one server? No. We’re running multiple, in a round-robin load-balanced farm. This means that all this refreshing impacted all of our servers, making the site as slow as a snail. And recover again. And go almost down again…. and come up again. Exactly what we’ve been seeing today.

What are we doing to prevent this from happening in the future?

The truths of offering a cloud service are: hardware fails, software has bugs and people make mistakes. Hardware failures are mitigated by the Windows Azure system. Software bugs? It seems we’re proficient at that. And yes, that was a giant mistake of us. Our job is to mitigate all of these and provide a reliable, robust service to you. We’ll be using the lessons learned today to improve our service, your service.

First of all, we’ve fixed our rookie mistake. This should have never happened and we’ll make sure that our code base is checked and repaired for faulty Equals / GetHashCode implementations.

Next, we’ll be reviewing our retry logic. Blindly retrying on any exception type isn’t the smartest thing to do, so it seems. We’ll be making sure the retry logic only fires on transient exceptions and not blindly on any exception that occurs.

We’ll also be investigating additional monitoring. We’re not sure yet about possible tools or services there, we do want to know when something is wrong and our CPU is heating the entire datacenter at 100%.

We're sorry!

Happy packaging!

The MyGet team

Pushing packages from MyGet to NuGet (or another feed)

Imagine you have a package repository hosted on MyGet. Every time a team member of your open source or enterprise project commits source code changes, your build server pushes an updated release to this package repository in the form of a prerelease NuGet package. Now what happens if a release to the official NuGet package source has to be created? Typically, you will either create a fresh package which will be the package to release, or download a package from your build server, change the version and upload that one to NuGet.org (or another repository). No need for such overloaded process anymore: MyGet will perform the push for you.

Setting up a continuous integration (CI) feed

First of all, you will need a CI feed to which your build server can push every NuGet package related to your project. Simply create your feed on MyGet, a three-second process. After creating your feed, MyGet will present you the feed URL as well as an API key. Optionally, you can make it a private feed and ensure only people who were granted the correct privileges can access your feed.

Next, configure your build server to push the NuGet artifacts to MyGet. Using TeamCity, for example, this can be done by adding a NuGet Push build step:

MyGet feed TeamCity

Pushing packages from MyGet to NuGet (or another feed)

The first time you want to push a package to another NuGet feed, you’ll probably have to configure the other feed’s URL and API key to use when pushing there. The push package feature is based on the package source proxy feature released earlier. Navigate to your feed and on the left hand side, click the “Package Sources” item and either add a new package source or edit the existing, default NuGet package source. In order to be able to push a package to another feed, the API key has to be specified. You can enter your NuGet.org (or another feed) API key and click the Save button.

MyGet package source selection

Pushing packages from MyGet to NuGet or any other feed is easy. From the moment a package source has been configured, using the “Push” button will enable you to push packages to another feed.

image

After clicking the “Push” button, MyGet will present you with an overview of the package which will be pushed to another feed. Select the feed to which you want to push and verify the other fields. When pushing a prerelease package to a stable package, simply make the Prerelease tag field blank. You can also modify the prerelease tag if wanted. If you do intend to push a prerelease version, simply continue clicking the “Push” button.

Push a NuGet package

Told you it was easy. MyGet will now push the package to the selected feed and inform you by e-mail should anything go wrong. Happy packaging!