MyGet Update: version 1.3.0

We are happy to announce a new version of the MyGet Web site containing a set of improvements, fixes and updates.

The following work items have been addressed in this release:

Product

  • number of plans has been reduced. We now have Free, Starter (formerly Small plan), Professional (formerly Large plan) and a new Enterprise plan
  • introduction of the new Enterprise plan focused on companies that require more performance, integration and security

Usability

  • improved layout and readability on smaller screen resolutions
  • auto-lower cased routing for feed URL (helps avoiding issues caused by incorrect casing of feed identifier)
Performance
  • upgraded to MVC4
Features
  • new identity provider: StackExchange
  • upgraded SymbolSource integration to take benefit from their improved and freshly released API
  • Gallery feeds now support a README in Markdown format, available in the feed's Gallery Settings
  • Enterprise plan: upgraded administrative & quota management dashboard
More details about the new features will be covered in follow-up blog posts soon.
 
Happy packaging!

Author: Maarten Balliauw on 24 Jul 2012

Planned maintenance July 26, 2012

We are planning a small window of maintenance on Thursday July 26, between 1:00pm and 2:00pm CET, during which we will roll out an update to the MyGet Web site.

During that timeframe, we will disable any synchronization we have in place with SymbolSource. Synchronization will pick up after our deployment.

We expect no other interruptions in service during this time frame, except for maybe a few seconds when switching VIPs.

Stay tuned!

Author: Maarten Balliauw on 24 Jul 2012

Site issues on July 2nd, 2012

What a day! July 2nd seemed to go from sunshine to rain in a couple of hours. The good news is: we’re back. Let’s first go through what happened…

What happened?

imageThis morning, you've probably received an e-mail stating that "Your Free subscription at MyGet has expired". Free? Expired? Even we were puzzled! We’ve quickly sent out an e-mail stating free subscriptions don’t expire and that we were investigating the issue. In our rush to find out what happened, we even forgot to remove some template text from that e-mail. Professional of us, really.

A couple of hours later, the website started acting funny: one moment it was up, one moment it appeared down. The strange thing was: we did not see this happening in our server monitoring page. The brand new Windows Azure portal was all shiny and bright, all systems running. A short glitch? Probably.

And then 4:00 PM (GMT+1) came around: we started to receive SMS messages from Pingdom. UP, DOWN, UP, DOWN. Some sweaty hours later, we’ve found the root cause of all this mess and shortly thereafter, deployed a hotfix to the cloud. What a day!

All-in-all, we’ve spammed you in the morning, had some small glitches during the day and went bezerk between 4:00 PM (GMT+1) and 7:00 PM (GMT+1).

What was wrong?

During the entire day, we were suspecting two of our latest changes. First of all, we’ve moved storage accounts in Windows Azure to make use of some of the new features in there with regards to mirroring packages. The code isn’t done yet, but we did already switch the storage accounts. Next to that, we’ve recently deployed our site using ASP.NET MVC 4 (RC). Was any of this the cause? We’ve been searching… But no.

Elmah revealed some interesting details. We were getting throttled on those new storage accounts. Was this due to the fact that we enabled storage analytics? We’ve disabled them and found that we were back up in full glory! For a moment, because this wasn’t the reason we were being throttled…

We process several operations asynchronously, such as updating our caches. It seemed that there were 16.000 commands in there, all telling the system to update the cache for one specific feed. One of our newest MyGet members managed to break a couple of gears there! Something we don’t judge. In contrary: there’s no better stress test or user test than real users.

A rookie mistake

Now why were there 16.000 messages unprocessed and why were some of them on the dead-letter queue? And why were our machines looking fine (all green!) while the website did not respond? The answer lies in the following piece of code:

1 [DebuggerDisplay("{Title} Version={Version}")] 2 public class FeedPackage 3 : TableServiceEntity 4 { 5 public string Id { get; set; } 6 public string Version { get; set; } 7 8 // ... 9 10 public override bool Equals(object obj) 11 { 12 if (ReferenceEquals(null, obj)) return false; 13 if (ReferenceEquals(this, obj)) return true; 14 if (obj.GetType() != typeof(FeedPackage)) return false; 15 return GetHashCode() == obj.GetHashCode(); 16 } 17 18 public override int GetHashCode() 19 { 20 int hashCode = 7; 21 22 if (PartitionKey != null) hashCode ^= PartitionKey.GetHashCode(); 23 if (RowKey != null) hashCode ^= RowKey.GetHashCode(); 24 25 return hashCode; 26 } 27 }

Did you spot it? We’ve gone with a complete rookie implementation of Equals and GetHashCode there…

The devil is in the fact that we’re using the hashcode of package A and compare it with package B. That’s a serious problem, as those hash codes are based on the hashcodes of their package identifier and package version. Guess what MSDN tells us

If two string objects are equal, the GetHashCode method returns identical values. However, there is not a unique hash code value for each unique string value. Different strings can return the same hash code.

Our newest member was in the position where he had a lot of packages uploaded to MyGet and by coincidence, two packages appeared to be “equal” using our faulty code above.

Now why did the website choke on that?

Windows Azure table storage, which we use to store all your data, In the .NET space, the WIndows Azure storage client uses WCF Data Service to fetch data. Deep in that framework, the Equals'() method is called on every entity to check whether a specific entity is already being tracked by the data context in use. And since we had a duplicate result for Equals(), an unhandled exception was thrown there.

Wait a minute: you guys don’t catch unhandled exceptions? Yes we do. And when such exception occurs, we retry some operations for up to five times. Retry operations? Well, rather than failing immediately, a good pattern is to retry a failing operation to check that the error wasn’t just a minor glitch like a network fault. On Windows Azure, it is recommended to do this for calls to any external system, like a webserver calling storage. Check David Aiken’s blog for some info on this.

So, retries… Yes. Retry an I/O operation with minor latency for five times. Have a user who’s experiencing an issue with the website F5 a couple of times. And have that queue with 16.000 operations pending hammer this storage account again. What will happen? Throttling. Windows Azure tells our system to retry again after a second. And our retry logic does exactly that. This retry logic causes some threads to sleep, all the queue processing and F5-ing causes some more thread to sleep and what happens? The server comes to a halt while it still looks okay to our external monitoring.

Are we running just one server? No. We’re running multiple, in a round-robin load-balanced farm. This means that all this refreshing impacted all of our servers, making the site as slow as a snail. And recover again. And go almost down again…. and come up again. Exactly what we’ve been seeing today.

What are we doing to prevent this from happening in the future?

The truths of offering a cloud service are: hardware fails, software has bugs and people make mistakes. Hardware failures are mitigated by the Windows Azure system. Software bugs? It seems we’re proficient at that. And yes, that was a giant mistake of us. Our job is to mitigate all of these and provide a reliable, robust service to you. We’ll be using the lessons learned today to improve our service, your service.

First of all, we’ve fixed our rookie mistake. This should have never happened and we’ll make sure that our code base is checked and repaired for faulty Equals / GetHashCode implementations.

Next, we’ll be reviewing our retry logic. Blindly retrying on any exception type isn’t the smartest thing to do, so it seems. We’ll be making sure the retry logic only fires on transient exceptions and not blindly on any exception that occurs.

We’ll also be investigating additional monitoring. We’re not sure yet about possible tools or services there, we do want to know when something is wrong and our CPU is heating the entire datacenter at 100%.

We're sorry!

Happy packaging!

The MyGet team

Author: Maarten Balliauw on 02 Jul 2012

Manage NuGet Package Sources using PowerShell

If you regularly need to manage your NuGet package sources in Visual Studio, and got familiar to using the NuGet Package Manager Console for all kinds of operations and automation, you might find the following approach appealing.

This approach will save you a few clicks and is based on a very cool tools package called ManagePackageSource. This NuGet package installs a few extra PowerShell commands into your NuGet PowerShell profile. That’s right, NuGet supports its own PowerShell profile! So instead of installing these tools packages over and over again every single time you create a new solution, let’s simply install these commands once and for all, so they are always available.

To do so, simply run the following command:

Install-Package ManagePackageSource

You should see the following output in the NuGet Package Manager Console

2012-05-30_2306

 

After that, you can simply uninstall the package again from your project to undo the changes in your working directory. The commands have been installed into your NuGet PowerShell profile and the profile has been reloaded, so they will remain there even without ever needing to install this package again.

If you happened to have another instance of Visual Studio open before installing this package, you can reload the profile as well in this other instance by running the following command in the NuGet Package Manager Console:

. $profile

In case you’re wondering where we changed stuff on your computer, you can find the NuGet_powershell.ps1 profile in the C:\Users\username\My Documents\WindowsPowerShell\ directory, and the new ManagePackageSource module can be found under the Modules directory inside that one.

To register a new NuGet feed into your Visual Studio, for instance a MyGet feed, you can run the following command now:

Add-PackageSource "My MyGet feed" "http://www.myget.org/F/myfeedname/"

If you're interested into the sources, you can obviously unzip the NuGet package or browse the code on our GitHub repository. Feel free to contribute your own extensions and improvements!

Author: Maarten Balliauw on 06 Jun 2012

MyGet support for multiple logins (and GitHub)

Last week, we’ve deployed a major update for MyGet. Next to support for pushing packages to other feeds and the new MyGet Gallery, we’ve done some updates to our authentication mechanisms. From now on, you can link multiple social identities to your MyGet account. For example, I’ve linked Google, Facebook and Windows Live ID to my own account so I don’t have to think about which account I have to use to login.

MyGet GitHub SupportAs a bonus we now also support GitHub as an identity provider. This enables you to log in to your GitHub account and be logged in across all GitHub related websites using a single sign-on experience.

You can set up these multiple identities (and GitHub) on your profile page. Follow the Add identity provider hyperlink and complete the process.

image

Happy packaging!

Author: Maarten Balliauw on 06 Jun 2012

Introducing MyGet Gallery

Included in our latest release of www.myget.org last week, we’ve deployed the new MyGet gallery. We’ve noticed a lot of NuGet packages for popular open-source projects (such as Dotspatial or Mvccontrib) are hidden in their continuous integration feeds on MyGet. To make it easier to find these hidden treasures, we’ve released the MyGet Gallery.

The MyGet Gallery enables you to browse around through selected public feeds and browse packages listed on those feeds. It’s an initial release of this feature and we’re looking forward to hearing your comments through our UserVoice.

image

Get your feed listed, too!

You can opt-in to be listed in the MyGet Gallery too. Simply navigate to your feeds and edit the settings under the MyGet Gallery tab. You can select to be either listed or unlisted in the MyGet Gallery. It’s also possible to add a logo of choice.

image

Happy packaging!

Author: Maarten Balliauw on 06 Jun 2012

Pushing packages from MyGet to NuGet (or another feed)

Imagine you have a package repository hosted on MyGet. Every time a team member of your open source or enterprise project commits source code changes, your build server pushes an updated release to this package repository in the form of a prerelease NuGet package. Now what happens if a release to the official NuGet package source has to be created? Typically, you will either create a fresh package which will be the package to release, or download a package from your build server, change the version and upload that one to NuGet.org (or another repository). No need for such overloaded process anymore: MyGet will perform the push for you.

Setting up a continuous integration (CI) feed

First of all, you will need a CI feed to which your build server can push every NuGet package related to your project. Simply create your feed on MyGet, a three-second process. After creating your feed, MyGet will present you the feed URL as well as an API key. Optionally, you can make it a private feed and ensure only people who were granted the correct privileges can access your feed.

Next, configure your build server to push the NuGet artifacts to MyGet. Using TeamCity, for example, this can be done by adding a NuGet Push build step:

MyGet feed TeamCity

Pushing packages from MyGet to NuGet (or another feed)

The first time you want to push a package to another NuGet feed, you’ll probably have to configure the other feed’s URL and API key to use when pushing there. The push package feature is based on the package source proxy feature released earlier. Navigate to your feed and on the left hand side, click the “Package Sources” item and either add a new package source or edit the existing, default NuGet package source. In order to be able to push a package to another feed, the API key has to be specified. You can enter your NuGet.org (or another feed) API key and click the Save button.

MyGet package source selection

Pushing packages from MyGet to NuGet or any other feed is easy. From the moment a package source has been configured, using the “Push” button will enable you to push packages to another feed.

image

After clicking the “Push” button, MyGet will present you with an overview of the package which will be pushed to another feed. Select the feed to which you want to push and verify the other fields. When pushing a prerelease package to a stable package, simply make the Prerelease tag field blank. You can also modify the prerelease tag if wanted. If you do intend to push a prerelease version, simply continue clicking the “Push” button.

Push a NuGet package

Told you it was easy. MyGet will now push the package to the selected feed and inform you by e-mail should anything go wrong. Happy packaging!

Author: Maarten Balliauw on 06 Jun 2012

NuGet $version$ token explained

Many questions that often come to mind when building NuGet packages are related to versioning. There's one question in particular I'd like to post here because it's one of the easier to answer. The question is: How do I use the $version$ token in the NuGet manifest (nuspec) file? Where does it get the version number from?

If you look at the NuGet docs explaining the nuspec replacement tokens, it states that the following - if it doesn't, my pull request got accepted :) - The assembly version as specified by the assembly's AssemblyVersionAttribute.

That is not entirely accurate (got a pull request accepted on the docs, so this should be fixed soonish). Consider the following AssemblyInfo.cs file contents for example.

[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]
[assembly: AssemblyInformationalVersion("1.0.2.0")]

This is where most people get confused. I won’t get into the details of which version attribute you should or should not use, as there can be good reasons to use either one of them in different scenarios. I’ll focus on how nuget is using this information to provide a version number to the $version$ replacement token in the nuspec file.

Building a NuGet package using a tokenized nuspec file that relies on assembly information can be achieved in various ways, for instance:

  1. nuget spec <csproj> to generate the tokenized nuspec, followed by nuget pack <csproj>
  2. nuget spec –a <assemblyPath> inside the csproj folder to generate a non-tokenized nuspec (so with the metadata already filled in), followed by nuget pack <nuspec>

Basically, it comes down to this:

  • If the AssemblyInformationalVersion attribute is available, then that one is used.
  • If the AssemblyInformationalVersion attribute is not available, then the AssemblyVersion attribute is used.
  • If none of the above are specified, your assembly will have a version number of 0.0.0.0, as well as the resulting package.
  • NuGet totally ignores the AssemblyFileVersion attribute.

Note that this behavior is the same when skipping the nuspec at all and building a nuget package directly from an assembly, using using nuget pack <assembly.dll>.

Author: Maarten Balliauw on 27 Apr 2012