MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Add packages from GitHub, BitBucket and CodePlex using MyGet build services

We’re pleased to announce some new features to MyGet build services. This feature allows you to add packages to your MyGet feed from any Git, Mercurial (hg) or Subversion repository out there. We’ll grab the sources, compile, package and make sure the result is listed on your feed. While still in beta, the feature is starting to take shape. In our latest release, we’ve shipped some interesting new features related to build services.

From your feed details page, you can navigate to build services. The “Add build source…” button has been around since the start and allows you to enter details of the source code repository manually. Because that can be a bit clumsy and since most public source code repositories out there have API’s available, we now support linking GitHub, BitBucket and CodePlex repositories with the click of a button!

image

After clicking one of these, you’ll be redirected to the code hosting provider for login.

image

After that, you can just check the projects you wish to add and link them automatically.

image

Apart from being able to easily link projects from GitHub, BitBucket and CodePlex, we also improved the build server itself a bit:

  • Support for portable libraries
  • TypeScript SDK 0.8.1.1 has been deployed
  • Private repositories on BitBucket now can be referenced
  • Git repositories with submodules can now be built (do note the submodule must be available over HTTP/HTTPS)

Happy packaging!

How we push GoogleAnalyticsTracker to NuGet

If you check Maarten’s blog post Tracking API usage with Google Analytics, you’ll see that a small open-source component evolved from MyGet. This component, GoogleAnalyticsTracker, lives on GitHub and NuGet and has since evolved into something that supports Windows Phone and Windows RT as well. Here’s his guest post:


It’s funny how things evolve. GoogleAnalyticsTracker started as a small component inside MyGet, and since a couple of weeks it uses MyGet to publish itself to NuGet. Say what? In this blog post, I’ll elaborate a bit on the development tools used on this tiny component.

Source code

Source code for GoogleAnalyticsTracker can be found on GitHub. This is the main entry point to all activity around this “project”. If you have a nice addition, feel free to fork it and send me a pull request.

Staging NuGet packages

Whenever I update the source code, I want to automatically build it and publish NuGet packages for it. Not directly to NuGet: I want to keep the regular version, the WinRT and WP version more or less in sync regarding version numbers. Also, I sometimes miss something which I fix again 5 minutes after. In the meanwhile, I like to have the generated package on some sort of “staging” feed, at MyGet. It’s even public, check http://www.myget.org/F/githubmaarten if you want to use my development artifacts.

When I decide it’s time for these packages to move to the “official NuGet package repository” at NuGet.org, I simply click the “push” button in my MyGet feed. Yes, that’s a manual step but I wanted to have some “gate” in the middle where I should explicitly do something. Here’s what happens after clicking “push”:

Push to NuGet

That’s right! You can use MyGet as a staging feed and from there push your packages onwards to any other feed out there. MyGet takes care of the uploading.

Building the package

There’s one thing which I forgot… How do I build these packages? Well… I don’t. I let MyGet Build Services.do the heavy lifting. On your feed, you can simply click the “Add GitHub project” button and a list of all your GitHub repos will be shown:

Build GitHub project

Tick a box and you’re ready to roll. And if you look carefully, you’ll see a “Build hook URL” being shown:

MyGet build hook

Back on GitHub, there’s this concept of “service hooks”, basically small utilities that you can fire whenever a new commit occurs on your repository. Wouldn’t it be awesome to trigger package creation on MyGet whenever I check in code to GitHub? Guess what…

GitHub build hook

That’s right! And MyGet even runs unit tests. Some sort of a continuous integration where I have the choice to promote packages to NuGet whenever I think they are stable.

MyGet Build Services - Public Beta

We’re happy to announce that MyGet Build Services is in Public Beta since, well, now. MyGet Build Services enable you to add packages to your feed by just giving us your Git, Mercurial or Subversion repo. We build it, we package it, we publish it. And starting today, every MyGet user can use it.

More information on MyGet Build Services can be found in a previous blog post, although some things have been updated. You’ve provided us feedback through our UserVoice, resulting in the following changes:

  • We now support .NET Framework 4.5. New features available in .NET Framework 4.5 are described at http://msdn.microsoft.com/en-us/library/vstudio/ms171868.aspx.
  • Build hooks! The list of your build definitions now shows a build hook URL which you can link to your GitHub or BitBucket repository. Calling this build hook (HTTP POST) will trigger a new build. Or in other words: a commit will trigger a build.
  • Our build server now checks for a file called build.bat first, then MyGet.sln, then your .sln and then .csproj/.vbproj files (UV)
  • We now support Git, Mercurial and Subversion repositories (UV)
  • You’ll now have some tools available in the environment variables (UV)
  • There’s more information in the build logs (UV)

Keep the feedback coming! Build Services are now enabled on every MyGet account. Do note MyGet Build Services are still a beta, and as Willy Wonka would say… “Little surprises around every corner, but nothing dangerous.”

Happy packaging!

PS: Why not deploy your freshly built apps using Octopus Deploy?

MyGet Build Services - Join the private beta!

Good news! Over the past 4 weeks we’ve been sending out tweets about our secret project MyGet project “wonka”. Today is the day Wonka shows his great stuff to the world… In short: MyGet Build Services enable you to add packages to your feed by just giving us your GitHub repo. We build it, we package it, we publish it.

Our build server searches for a file called MyGet.sln and builds that. No problem if it's not there: we'll try and build other projects then. We'll run unit tests (NUnit, XUnit, MSTest and some more) and fail when those fail. We'll search for packages generated by your solution and if none are generated, we take a wild guess and create them for you.

To make it more visual, here are some screenshots. First, you have to add a build source, for example a GitHub repository (in fact, GitHub is all we currently support):

MyGet Add build source

After that, you simply click “Build”. A couple of seconds or minutes later, your fresh package is available on your feed:

MyGet build package

MyGet package result

If you want to see what happened, the build log is available for review as well:

MyGet build log

Enroll now!

Starting today, you can enroll for our private beta. You’ll get on a waiting list and as we improve build capacity, you will be granted access to the beta. If you’re in, tell us how it behaves. What works, what doesn’t, what would you like to see improved. Enroll for this private beta now via http://www.myget.org/buildservices. Limited seats!

Do note it’s still a beta, and as Willy Wonka would say… “Little surprises around every corner, but nothing dangerous.”

Happy packaging!

PS: Why not deploy your freshly built apps using Octopus Deploy?

Site issues on July 2nd, 2012

What a day! July 2nd seemed to go from sunshine to rain in a couple of hours. The good news is: we’re back. Let’s first go through what happened…

What happened?

imageThis morning, you've probably received an e-mail stating that "Your Free subscription at MyGet has expired". Free? Expired? Even we were puzzled! We’ve quickly sent out an e-mail stating free subscriptions don’t expire and that we were investigating the issue. In our rush to find out what happened, we even forgot to remove some template text from that e-mail. Professional of us, really.

A couple of hours later, the website started acting funny: one moment it was up, one moment it appeared down. The strange thing was: we did not see this happening in our server monitoring page. The brand new Windows Azure portal was all shiny and bright, all systems running. A short glitch? Probably.

And then 4:00 PM (GMT+1) came around: we started to receive SMS messages from Pingdom. UP, DOWN, UP, DOWN. Some sweaty hours later, we’ve found the root cause of all this mess and shortly thereafter, deployed a hotfix to the cloud. What a day!

All-in-all, we’ve spammed you in the morning, had some small glitches during the day and went bezerk between 4:00 PM (GMT+1) and 7:00 PM (GMT+1).

What was wrong?

During the entire day, we were suspecting two of our latest changes. First of all, we’ve moved storage accounts in Windows Azure to make use of some of the new features in there with regards to mirroring packages. The code isn’t done yet, but we did already switch the storage accounts. Next to that, we’ve recently deployed our site using ASP.NET MVC 4 (RC). Was any of this the cause? We’ve been searching… But no.

Elmah revealed some interesting details. We were getting throttled on those new storage accounts. Was this due to the fact that we enabled storage analytics? We’ve disabled them and found that we were back up in full glory! For a moment, because this wasn’t the reason we were being throttled…

We process several operations asynchronously, such as updating our caches. It seemed that there were 16.000 commands in there, all telling the system to update the cache for one specific feed. One of our newest MyGet members managed to break a couple of gears there! Something we don’t judge. In contrary: there’s no better stress test or user test than real users.

A rookie mistake

Now why were there 16.000 messages unprocessed and why were some of them on the dead-letter queue? And why were our machines looking fine (all green!) while the website did not respond? The answer lies in the following piece of code:

1 [DebuggerDisplay("{Title} Version={Version}")] 2 public class FeedPackage 3 : TableServiceEntity 4 { 5 public string Id { get; set; } 6 public string Version { get; set; } 7 8 // ... 9 10 public override bool Equals(object obj) 11 { 12 if (ReferenceEquals(null, obj)) return false; 13 if (ReferenceEquals(this, obj)) return true; 14 if (obj.GetType() != typeof(FeedPackage)) return false; 15 return GetHashCode() == obj.GetHashCode(); 16 } 17 18 public override int GetHashCode() 19 { 20 int hashCode = 7; 21 22 if (PartitionKey != null) hashCode ^= PartitionKey.GetHashCode(); 23 if (RowKey != null) hashCode ^= RowKey.GetHashCode(); 24 25 return hashCode; 26 } 27 }

Did you spot it? We’ve gone with a complete rookie implementation of Equals and GetHashCode there…

The devil is in the fact that we’re using the hashcode of package A and compare it with package B. That’s a serious problem, as those hash codes are based on the hashcodes of their package identifier and package version. Guess what MSDN tells us

If two string objects are equal, the GetHashCode method returns identical values. However, there is not a unique hash code value for each unique string value. Different strings can return the same hash code.

Our newest member was in the position where he had a lot of packages uploaded to MyGet and by coincidence, two packages appeared to be “equal” using our faulty code above.

Now why did the website choke on that?

Windows Azure table storage, which we use to store all your data, In the .NET space, the WIndows Azure storage client uses WCF Data Service to fetch data. Deep in that framework, the Equals'() method is called on every entity to check whether a specific entity is already being tracked by the data context in use. And since we had a duplicate result for Equals(), an unhandled exception was thrown there.

Wait a minute: you guys don’t catch unhandled exceptions? Yes we do. And when such exception occurs, we retry some operations for up to five times. Retry operations? Well, rather than failing immediately, a good pattern is to retry a failing operation to check that the error wasn’t just a minor glitch like a network fault. On Windows Azure, it is recommended to do this for calls to any external system, like a webserver calling storage. Check David Aiken’s blog for some info on this.

So, retries… Yes. Retry an I/O operation with minor latency for five times. Have a user who’s experiencing an issue with the website F5 a couple of times. And have that queue with 16.000 operations pending hammer this storage account again. What will happen? Throttling. Windows Azure tells our system to retry again after a second. And our retry logic does exactly that. This retry logic causes some threads to sleep, all the queue processing and F5-ing causes some more thread to sleep and what happens? The server comes to a halt while it still looks okay to our external monitoring.

Are we running just one server? No. We’re running multiple, in a round-robin load-balanced farm. This means that all this refreshing impacted all of our servers, making the site as slow as a snail. And recover again. And go almost down again…. and come up again. Exactly what we’ve been seeing today.

What are we doing to prevent this from happening in the future?

The truths of offering a cloud service are: hardware fails, software has bugs and people make mistakes. Hardware failures are mitigated by the Windows Azure system. Software bugs? It seems we’re proficient at that. And yes, that was a giant mistake of us. Our job is to mitigate all of these and provide a reliable, robust service to you. We’ll be using the lessons learned today to improve our service, your service.

First of all, we’ve fixed our rookie mistake. This should have never happened and we’ll make sure that our code base is checked and repaired for faulty Equals / GetHashCode implementations.

Next, we’ll be reviewing our retry logic. Blindly retrying on any exception type isn’t the smartest thing to do, so it seems. We’ll be making sure the retry logic only fires on transient exceptions and not blindly on any exception that occurs.

We’ll also be investigating additional monitoring. We’re not sure yet about possible tools or services there, we do want to know when something is wrong and our CPU is heating the entire datacenter at 100%.

We're sorry!

Happy packaging!

The MyGet team

MyGet support for multiple logins (and GitHub)

Last week, we’ve deployed a major update for MyGet. Next to support for pushing packages to other feeds and the new MyGet Gallery, we’ve done some updates to our authentication mechanisms. From now on, you can link multiple social identities to your MyGet account. For example, I’ve linked Google, Facebook and Windows Live ID to my own account so I don’t have to think about which account I have to use to login.

MyGet GitHub SupportAs a bonus we now also support GitHub as an identity provider. This enables you to log in to your GitHub account and be logged in across all GitHub related websites using a single sign-on experience.

You can set up these multiple identities (and GitHub) on your profile page. Follow the Add identity provider hyperlink and complete the process.

image

Happy packaging!

Introducing MyGet Gallery

Included in our latest release of www.myget.org last week, we’ve deployed the new MyGet gallery. We’ve noticed a lot of NuGet packages for popular open-source projects (such as Dotspatial or Mvccontrib) are hidden in their continuous integration feeds on MyGet. To make it easier to find these hidden treasures, we’ve released the MyGet Gallery.

The MyGet Gallery enables you to browse around through selected public feeds and browse packages listed on those feeds. It’s an initial release of this feature and we’re looking forward to hearing your comments through our UserVoice.

image

Get your feed listed, too!

You can opt-in to be listed in the MyGet Gallery too. Simply navigate to your feeds and edit the settings under the MyGet Gallery tab. You can select to be either listed or unlisted in the MyGet Gallery. It’s also possible to add a logo of choice.

image

Happy packaging!

Pushing packages from MyGet to NuGet (or another feed)

Imagine you have a package repository hosted on MyGet. Every time a team member of your open source or enterprise project commits source code changes, your build server pushes an updated release to this package repository in the form of a prerelease NuGet package. Now what happens if a release to the official NuGet package source has to be created? Typically, you will either create a fresh package which will be the package to release, or download a package from your build server, change the version and upload that one to NuGet.org (or another repository). No need for such overloaded process anymore: MyGet will perform the push for you.

Setting up a continuous integration (CI) feed

First of all, you will need a CI feed to which your build server can push every NuGet package related to your project. Simply create your feed on MyGet, a three-second process. After creating your feed, MyGet will present you the feed URL as well as an API key. Optionally, you can make it a private feed and ensure only people who were granted the correct privileges can access your feed.

Next, configure your build server to push the NuGet artifacts to MyGet. Using TeamCity, for example, this can be done by adding a NuGet Push build step:

MyGet feed TeamCity

Pushing packages from MyGet to NuGet (or another feed)

The first time you want to push a package to another NuGet feed, you’ll probably have to configure the other feed’s URL and API key to use when pushing there. The push package feature is based on the package source proxy feature released earlier. Navigate to your feed and on the left hand side, click the “Package Sources” item and either add a new package source or edit the existing, default NuGet package source. In order to be able to push a package to another feed, the API key has to be specified. You can enter your NuGet.org (or another feed) API key and click the Save button.

MyGet package source selection

Pushing packages from MyGet to NuGet or any other feed is easy. From the moment a package source has been configured, using the “Push” button will enable you to push packages to another feed.

image

After clicking the “Push” button, MyGet will present you with an overview of the package which will be pushed to another feed. Select the feed to which you want to push and verify the other fields. When pushing a prerelease package to a stable package, simply make the Prerelease tag field blank. You can also modify the prerelease tag if wanted. If you do intend to push a prerelease version, simply continue clicking the “Push” button.

Push a NuGet package

Told you it was easy. MyGet will now push the package to the selected feed and inform you by e-mail should anything go wrong. Happy packaging!

Pro NuGet is finally there!

Short version: Install-Package ProNuget or http://amzn.to/pronuget

Pro NuGet - Continuous integration Package RestoreIt’s been a while since I wrote my first book. After I’ve been telling that writing a book is horrendous (try writing a chapter per week after your office hours…) and that I would never write on again, my partner-in-crime Xavier Decoster and I had the same idea at the same time: what about a book on NuGet? So here it is: Pro NuGet is fresh off the presses (or on Kindle).

Special thanks go out to Scott Hanselman and Phil Haack for writing our foreword. Also big kudos to all who’ve helped us out now and then and did some small reviews. Yes Rob, Paul, David, Phil, Hadi: that’s you guys.

Why a book on NuGet?

Why not? At the time we decided we would start writing a book (september 2011), NuGet was out there for a while already. Yet, most users then (and still today) were using NuGet only as a means of installing packages, some creating packages. But NuGet is much more! And that’s what we wanted to write about. We did not want to create a reference guide on what NuGet command were available. We wanted to focus on best practices we’ve learned over the past few months using NuGet.

Some scenarios covered in our book:

  • What’s the big picture on package management?
  • Flashback last week: NuGet.org was down. How do you keep your team working if you depend on that external resource?
  • Is it a good idea to auto-update NuGet packages in a continous integration process?
  • Use the PowerShell console in VS2010/11. How do I write my own NuGet PowerShell Cmdlets? What can I do in there?
  • Why would you host your own NuGet repository?
  • Using NuGet for continuous delivery
  • More!

I feel we’ve managed to cover a lot of concepts that go beyond “how to use NuGet vX” and instead have given as much guidance as possible. Questions, suggestions, remarks, … are all welcome. And a click on “Add to cart” is also a good idea ;-)

Where MyGet is going

Time flies. MyGet has been online for over half a year! We’ve started MyGet as a proof-of-concept which kept growing, thanks to you. You’re one of the more than 600 NuGet feeds we are currently hosting in two datacenters worldwide!

New user interface design, new blog

We’ve “moved your cheese around”. We’ve been redesigning our UI and added a few usability improvements here and there.

Next to that, our new blog went live at blog.myget.org. We will be using it to announce new features, provide sample use cases for MyGet and share other information with you.

More features

It have been a couple of busy weeks. The new release of MyGet which we've just deployed contains an important number of new features. Some small ones, such as the ability to delete all non-latest packages from a feed at once.
We've also added an often requested one: the ability to include packages in your MyGet feed from a different source such as Chocolatey or Orchard Gallery. Xavier blogged about this in MyGet tops Vanilla NuGet feeds with a Chocolatey flavor.

Building on that feature, you can link another feed and include a complete or filtered set of packages from that feed. This offers the ability to have one single package source endpoint which aggregates different external feeds and is enriched with your privately hosted packages. Maarten blogged about the why and how in his post on MyGet package source proxy (beta).

Let us know about your experiences with these new features. You posted these features on our UserVoice feedback forum, we are looking forward to hearing more at UserVoice!

Introducing subscriptions

Since we want to build much more features and improvements, we're looking for some funding to buy ourselves time to do that. We’re introducing subscriptions: a free plan, which still allows you to do what you’ve been doing today. Our small, medium and large plans offer some additional capabilities to your feed. No worries! We’ve promised you to host your feeds and we’ll keep doing that.

Checkout the available plans at www.myget.org/plans and discover which plan we've assigned to you for two months as a gift to say thank you for using MyGet.

It's your product

Know that you can reach out to us through UserVoice or info@myget.org! Let us know your thoughts, feature requests or anything else you would like to share! MyGet is your product, and we want to make sure it stays that way. We’re just the code monkeys making sure you can host your packages and dependencies in a reliable and useful way.

Happy packaging!

Maarten & Xavier
The MyGet Team