MyGet Build Services - Join the private beta!

Good news! Over the past 4 weeks we’ve been sending out tweets about our secret project MyGet project “wonka”. Today is the day Wonka shows his great stuff to the world… In short: MyGet Build Services enable you to add packages to your feed by just giving us your GitHub repo. We build it, we package it, we publish it.

Our build server searches for a file called MyGet.sln and builds that. No problem if it's not there: we'll try and build other projects then. We'll run unit tests (NUnit, XUnit, MSTest and some more) and fail when those fail. We'll search for packages generated by your solution and if none are generated, we take a wild guess and create them for you.

To make it more visual, here are some screenshots. First, you have to add a build source, for example a GitHub repository (in fact, GitHub is all we currently support):

MyGet Add build source

After that, you simply click “Build”. A couple of seconds or minutes later, your fresh package is available on your feed:

MyGet build package

MyGet package result

If you want to see what happened, the build log is available for review as well:

MyGet build log

Enroll now!

Starting today, you can enroll for our private beta. You’ll get on a waiting list and as we improve build capacity, you will be granted access to the beta. If you’re in, tell us how it behaves. What works, what doesn’t, what would you like to see improved. Enroll for this private beta now via http://www.myget.org/buildservices. Limited seats!

Do note it’s still a beta, and as Willy Wonka would say… “Little surprises around every corner, but nothing dangerous.”

Happy packaging!

PS: Why not deploy your freshly built apps using Octopus Deploy?

Author: Maarten Balliauw on 22 Aug 2012

Feature highlight: MyGet Gallery

It's been less than two months since we introduced the public MyGet Gallery, and we are very happy to have some awesome projects onboard, often shipping development builds or using MyGet as a staging environment before pushing packages upstream. These feeds deserve more attention and people should know these feeds exist.

Did you know you can get the latest development builds of SignalR and MVC Contrib delivered through MyGet?

If you think your project's development builds deserve a place in our Gallery, you can easily opt-in using your feed's Gallery Settings.

Gallery feed readme

Our latest v1.3 release now allows feed owners to add a feed readme for publication in the MyGet Gallery. This way, feed owners now can easily add some instructions or disclaimers to the gallery feed details page using the well-known markdown format. Don't forget to add a link to your project page and/or sources as well.

You might recognize the pagedown markdown editor there, which is also used on the StackExchange Web sites, so you are probably already familiar with it. In return for this excellent control, we also added the StackOverflow identity provider to our login page. People who prefer to login using their StackOverflow account can now do so, or you can link it to your other identities within your MyGet account as well.

Layout improvements

While at it and based on the invaluable feedback of our users, we also improved the look-n-feel of the packages and feed listings by removing clutter and putting focus on content.

You can now not only see which packages are listed, and what version they're at, but you can also track when this version has become available. A simple click on the icon next to the package will trigger a package download in your browser, whilst the icons on the top right bring you to the SymbolSource symbols repository of this feed (if available) or give access to the RSS feed associated with this package repository.

What's next?

The screenshot below (taken from our latest development builds) is giving you a glimpse at what's coming next: user profiles and activity streams! That's right: you'll be able to get insights into the activity of a public feed or track the history of a given package. Coming soon...

We hope you like it! Keep those suggestions and feedback coming! Who knows, maybe one day you'll see your wish granted :)

Happy packaging!

Author: Maarten Balliauw on 25 Jul 2012

MyGet Update: version 1.3.0

We are happy to announce a new version of the MyGet Web site containing a set of improvements, fixes and updates.

The following work items have been addressed in this release:

Product

  • number of plans has been reduced. We now have Free, Starter (formerly Small plan), Professional (formerly Large plan) and a new Enterprise plan
  • introduction of the new Enterprise plan focused on companies that require more performance, integration and security

Usability

  • improved layout and readability on smaller screen resolutions
  • auto-lower cased routing for feed URL (helps avoiding issues caused by incorrect casing of feed identifier)
Performance
  • upgraded to MVC4
Features
  • new identity provider: StackExchange
  • upgraded SymbolSource integration to take benefit from their improved and freshly released API
  • Gallery feeds now support a README in Markdown format, available in the feed's Gallery Settings
  • Enterprise plan: upgraded administrative & quota management dashboard
More details about the new features will be covered in follow-up blog posts soon.
 
Happy packaging!

Author: Maarten Balliauw on 24 Jul 2012

Planned maintenance July 26, 2012

We are planning a small window of maintenance on Thursday July 26, between 1:00pm and 2:00pm CET, during which we will roll out an update to the MyGet Web site.

During that timeframe, we will disable any synchronization we have in place with SymbolSource. Synchronization will pick up after our deployment.

We expect no other interruptions in service during this time frame, except for maybe a few seconds when switching VIPs.

Stay tuned!

Author: Maarten Balliauw on 24 Jul 2012

Site issues on July 2nd, 2012

What a day! July 2nd seemed to go from sunshine to rain in a couple of hours. The good news is: we’re back. Let’s first go through what happened…

What happened?

imageThis morning, you've probably received an e-mail stating that "Your Free subscription at MyGet has expired". Free? Expired? Even we were puzzled! We’ve quickly sent out an e-mail stating free subscriptions don’t expire and that we were investigating the issue. In our rush to find out what happened, we even forgot to remove some template text from that e-mail. Professional of us, really.

A couple of hours later, the website started acting funny: one moment it was up, one moment it appeared down. The strange thing was: we did not see this happening in our server monitoring page. The brand new Windows Azure portal was all shiny and bright, all systems running. A short glitch? Probably.

And then 4:00 PM (GMT+1) came around: we started to receive SMS messages from Pingdom. UP, DOWN, UP, DOWN. Some sweaty hours later, we’ve found the root cause of all this mess and shortly thereafter, deployed a hotfix to the cloud. What a day!

All-in-all, we’ve spammed you in the morning, had some small glitches during the day and went bezerk between 4:00 PM (GMT+1) and 7:00 PM (GMT+1).

What was wrong?

During the entire day, we were suspecting two of our latest changes. First of all, we’ve moved storage accounts in Windows Azure to make use of some of the new features in there with regards to mirroring packages. The code isn’t done yet, but we did already switch the storage accounts. Next to that, we’ve recently deployed our site using ASP.NET MVC 4 (RC). Was any of this the cause? We’ve been searching… But no.

Elmah revealed some interesting details. We were getting throttled on those new storage accounts. Was this due to the fact that we enabled storage analytics? We’ve disabled them and found that we were back up in full glory! For a moment, because this wasn’t the reason we were being throttled…

We process several operations asynchronously, such as updating our caches. It seemed that there were 16.000 commands in there, all telling the system to update the cache for one specific feed. One of our newest MyGet members managed to break a couple of gears there! Something we don’t judge. In contrary: there’s no better stress test or user test than real users.

A rookie mistake

Now why were there 16.000 messages unprocessed and why were some of them on the dead-letter queue? And why were our machines looking fine (all green!) while the website did not respond? The answer lies in the following piece of code:

1 [DebuggerDisplay("{Title} Version={Version}")] 2 public class FeedPackage 3 : TableServiceEntity 4 { 5 public string Id { get; set; } 6 public string Version { get; set; } 7 8 // ... 9 10 public override bool Equals(object obj) 11 { 12 if (ReferenceEquals(null, obj)) return false; 13 if (ReferenceEquals(this, obj)) return true; 14 if (obj.GetType() != typeof(FeedPackage)) return false; 15 return GetHashCode() == obj.GetHashCode(); 16 } 17 18 public override int GetHashCode() 19 { 20 int hashCode = 7; 21 22 if (PartitionKey != null) hashCode ^= PartitionKey.GetHashCode(); 23 if (RowKey != null) hashCode ^= RowKey.GetHashCode(); 24 25 return hashCode; 26 } 27 }

Did you spot it? We’ve gone with a complete rookie implementation of Equals and GetHashCode there…

The devil is in the fact that we’re using the hashcode of package A and compare it with package B. That’s a serious problem, as those hash codes are based on the hashcodes of their package identifier and package version. Guess what MSDN tells us

If two string objects are equal, the GetHashCode method returns identical values. However, there is not a unique hash code value for each unique string value. Different strings can return the same hash code.

Our newest member was in the position where he had a lot of packages uploaded to MyGet and by coincidence, two packages appeared to be “equal” using our faulty code above.

Now why did the website choke on that?

Windows Azure table storage, which we use to store all your data, In the .NET space, the WIndows Azure storage client uses WCF Data Service to fetch data. Deep in that framework, the Equals'() method is called on every entity to check whether a specific entity is already being tracked by the data context in use. And since we had a duplicate result for Equals(), an unhandled exception was thrown there.

Wait a minute: you guys don’t catch unhandled exceptions? Yes we do. And when such exception occurs, we retry some operations for up to five times. Retry operations? Well, rather than failing immediately, a good pattern is to retry a failing operation to check that the error wasn’t just a minor glitch like a network fault. On Windows Azure, it is recommended to do this for calls to any external system, like a webserver calling storage. Check David Aiken’s blog for some info on this.

So, retries… Yes. Retry an I/O operation with minor latency for five times. Have a user who’s experiencing an issue with the website F5 a couple of times. And have that queue with 16.000 operations pending hammer this storage account again. What will happen? Throttling. Windows Azure tells our system to retry again after a second. And our retry logic does exactly that. This retry logic causes some threads to sleep, all the queue processing and F5-ing causes some more thread to sleep and what happens? The server comes to a halt while it still looks okay to our external monitoring.

Are we running just one server? No. We’re running multiple, in a round-robin load-balanced farm. This means that all this refreshing impacted all of our servers, making the site as slow as a snail. And recover again. And go almost down again…. and come up again. Exactly what we’ve been seeing today.

What are we doing to prevent this from happening in the future?

The truths of offering a cloud service are: hardware fails, software has bugs and people make mistakes. Hardware failures are mitigated by the Windows Azure system. Software bugs? It seems we’re proficient at that. And yes, that was a giant mistake of us. Our job is to mitigate all of these and provide a reliable, robust service to you. We’ll be using the lessons learned today to improve our service, your service.

First of all, we’ve fixed our rookie mistake. This should have never happened and we’ll make sure that our code base is checked and repaired for faulty Equals / GetHashCode implementations.

Next, we’ll be reviewing our retry logic. Blindly retrying on any exception type isn’t the smartest thing to do, so it seems. We’ll be making sure the retry logic only fires on transient exceptions and not blindly on any exception that occurs.

We’ll also be investigating additional monitoring. We’re not sure yet about possible tools or services there, we do want to know when something is wrong and our CPU is heating the entire datacenter at 100%.

We're sorry!

Happy packaging!

The MyGet team

Author: Maarten Balliauw on 02 Jul 2012

Manage NuGet Package Sources using PowerShell

If you regularly need to manage your NuGet package sources in Visual Studio, and got familiar to using the NuGet Package Manager Console for all kinds of operations and automation, you might find the following approach appealing.

This approach will save you a few clicks and is based on a very cool tools package called ManagePackageSource. This NuGet package installs a few extra PowerShell commands into your NuGet PowerShell profile. That’s right, NuGet supports its own PowerShell profile! So instead of installing these tools packages over and over again every single time you create a new solution, let’s simply install these commands once and for all, so they are always available.

To do so, simply run the following command:

Install-Package ManagePackageSource

You should see the following output in the NuGet Package Manager Console

2012-05-30_2306

 

After that, you can simply uninstall the package again from your project to undo the changes in your working directory. The commands have been installed into your NuGet PowerShell profile and the profile has been reloaded, so they will remain there even without ever needing to install this package again.

If you happened to have another instance of Visual Studio open before installing this package, you can reload the profile as well in this other instance by running the following command in the NuGet Package Manager Console:

. $profile

In case you’re wondering where we changed stuff on your computer, you can find the NuGet_powershell.ps1 profile in the C:\Users\username\My Documents\WindowsPowerShell\ directory, and the new ManagePackageSource module can be found under the Modules directory inside that one.

To register a new NuGet feed into your Visual Studio, for instance a MyGet feed, you can run the following command now:

Add-PackageSource "My MyGet feed" "http://www.myget.org/F/myfeedname/"

If you're interested into the sources, you can obviously unzip the NuGet package or browse the code on our GitHub repository. Feel free to contribute your own extensions and improvements!

Author: Maarten Balliauw on 06 Jun 2012

MyGet support for multiple logins (and GitHub)

Last week, we’ve deployed a major update for MyGet. Next to support for pushing packages to other feeds and the new MyGet Gallery, we’ve done some updates to our authentication mechanisms. From now on, you can link multiple social identities to your MyGet account. For example, I’ve linked Google, Facebook and Windows Live ID to my own account so I don’t have to think about which account I have to use to login.

MyGet GitHub SupportAs a bonus we now also support GitHub as an identity provider. This enables you to log in to your GitHub account and be logged in across all GitHub related websites using a single sign-on experience.

You can set up these multiple identities (and GitHub) on your profile page. Follow the Add identity provider hyperlink and complete the process.

image

Happy packaging!

Author: Maarten Balliauw on 06 Jun 2012

Introducing MyGet Gallery

Included in our latest release of www.myget.org last week, we’ve deployed the new MyGet gallery. We’ve noticed a lot of NuGet packages for popular open-source projects (such as Dotspatial or Mvccontrib) are hidden in their continuous integration feeds on MyGet. To make it easier to find these hidden treasures, we’ve released the MyGet Gallery.

The MyGet Gallery enables you to browse around through selected public feeds and browse packages listed on those feeds. It’s an initial release of this feature and we’re looking forward to hearing your comments through our UserVoice.

image

Get your feed listed, too!

You can opt-in to be listed in the MyGet Gallery too. Simply navigate to your feeds and edit the settings under the MyGet Gallery tab. You can select to be either listed or unlisted in the MyGet Gallery. It’s also possible to add a logo of choice.

image

Happy packaging!

Author: Maarten Balliauw on 06 Jun 2012