MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Downtime September 11 and 12, 2013 - Root cause

On September 11 and 12, 2013, we have experienced some downtime. The website and back-end services have crashed a number of times across all instances, bringing our system to a halt a couple of times. We hate downtime as much as you do and want to apologize for the trouble this may have caused you. Let’s have a look at the symptoms and root cause.

Symptoms

Twitter and e-mail notified us of this a good 2 minutes before our monitoring kicked in. When we had a look at our status dashboards, we saw high CPU usage on all machines, inexplicable crashes of machines and an error message in the Windows event logs right before the machines died and restarted.

In our back-end queue, where all work is added and waits to be processed, we found 25 messages that seemed to be retried over and over again, having been dequeued over 150 times.

We have manually removed these messages from the queue and inserted them again after the website was running stable again. After monitoring the situation for a couple of hours, all systems seemed to be running stable again. Seemed…

A number of hours later, we received some additional monitoring events. The problem was back! This time we decided not to fire fight but dig deeper into the issue we were facing. We took the site offline and connected to the machines to analyze event logs and so on. Except for the error message in the Windows event log, nothing was going on. Next, hooked a debugger to it, waiting for anything to happen.

The issue

Perhaps a root cause analysis is not the best moment to blog about our new features and updates in our latest deployment, but we have to in this case. Last Tuesday, we deployed our 1.8 release which brings an update to retention policies. These allow MyGet to automatically delete packages from a feed after a certain number of packages has been added, an automatic feed cleanup as it were.

The updated retention policies feature now tries not to break a dependency chain. Before, packages that were depended on by other packages but were subject to the retention policy settings would be deleted. Now, the retention policy handler respects the entire dependency chain for a package and will no longer remove depended packages. This, of course, requires us to parse the entire dependency tree for a package.

imageNow what happens if package A depends on package B, B depends on C and C depends on B? Right: that’s a potential infinite loop. Guess what was happening: a StackOverflowException, crashing our entire machine without giving it the chance to write output to our error logs.

Because of the crash, the message in our backend was requeued over and over. We typically move a message to a dead letter queue after a number of retries (typically 32). However because of the crash, that logic couldn’t kick in and move the message out, resulting in the message being requeued over and over again, triggering the same behavior again.

While we test our application thoroughly before going to production, this was an edge case. Who would create a package with circular dependencies anyway? We found that on one feed, a circular package dependency did exist.

Solution

We have made a fix to our algorithm which now supports this scenario and will stop following the circular dependency.

After confirming our algorithm, we have deployed it at 6 AM (CET) this morning, resolving the issue. The messages that were stuck in our queue (and triggered the initial issue) have been requeued to confirm correct working and  were all processed successfully.

Our status page can always be found at http://status.myget.org, showing uptime of our last months as well as the outage of the past hours.

Again, we do apologize for the inconvenience caused.

Happy packaging!

Online gaming with OCTGN and NuGet

OCTGN online card and table game networkWe’ve been looking through some of the feeds that are hosted on MyGet and have found a nice use case for both NuGet and MyGet: the open-source project OCTGN, the Online Card and Table Game Network. As a result of that, we’ve had a chat with Kelly Elton, developer and maintainer of OCTGN.

Can you tell us a little bit about OCTGN?

“The title of the game kind of summarizes this, but OCTGN is a gaming platform that allows users to play card and board games with each other on the Internet. After installation, gamers get a virtual table top where they can download additional games from a repository, chat with other gamers and compete against each other.”

We’re assuming that the games repository is in fact a NuGet feed, is that correct?

Yeah, that’s an interesting one. Started using NuGet a while ago and it occurred to me that you could use NuGet as an application update system. It has metadata in there which you can query and the NuGet.Core assembly gives you a nice library for working with it.

Originally, our users would download an OPC (Open Packaging Conventions) file which is essentially a ZIP file. A NuGet package essentially also is an OPC package, so we were already using some form of packaging for distributing games before. And then we did some rework and tore some pieces apart and realized we could start packaging using NuGet to get things like searchable metadata, a formal upgrade system and so on.”

A lot of the games are card games, are you embedding the card artwork in the NuGet packages as well?

“No. Sets of cards can be 100 MB to 200 MB in size. Not all players will use all cards all the time and we don’t want game updates to download all artwork when it hasn’t changed between versions. Also we have a card proxy which generates images from card data on the fly if they don't exist, which allows us to cram a game that used to require 100MB of images to play into a 10MB package, with the option to download the cards from another source if they so choose.

We developed a system that allows you to specify the metadata for the cards which is kind of a specialized markup language and then download the artwork on-the-fly, only when needed.”

Who is developing the games for OCTGN? How can someone contribute?

In-game screenshot”Most of the games are developed by community members. OCTGN has been around since 2006 and has changed hands a few times but it’s essentially run by the community. I’m currently doing most of the development but I have some people helping me. The games themselves are developed by people, using Python and XML. We have a builder (o8build) which comes with OCTGN, takes the Python code and XML files and packages everything up so our game developers don’t have to know too much about NuGet themselves. There’s some info about that on our GitHub space.

Contributions are done in several ways. We have different NuGet feeds that are managed by game owners, we also have our official feeds that contain games that we want to be part of our system. You get added to that feed based on different metrics like game quality, how active development is and so on. There are different feeds maintained by other members of the community and they can take on some games as well.

We only host the official feed, but since it's so easy to get setup with MyGet, people in the community have had no problems at all setting up their own feeds.

Are all feeds public feeds?

“Some of our game developers asked if it would be possible to make a private feed for their games, we’re now looking into adding support for that in our client. We weren’t sure how we could authenticate the user towards these private feeds as the NuGet client libraries aren’t that well documented. But I’ve seen there is an ICredentialProvider interface in there we can probably use.”

How has your experience with MyGet been so far?

“Very good, we’ve enjoyed it quite a bit. We didn’t quite know if it was going to be the right system for the job at first as we were almost going to abstract away the way packages were offered to the game. Our big concern was that we have almost 40.000 registered users and we didn’t want to step on your toes with regards to package sizes, bandwidth consumed and so on. We wanted to make sure we weren’t causing any cause of problems as a free user of MyGet.

As far as quality goes we have absolutely had no problems with MyGet at all. It’s really easy to upload and manage packages and we’re now looking into using the build system you have as that’s pretty neat.”

As a last question, do you have any advice for people starting with NuGet or MyGet?

“For MyGet not so much as it can’t be more straightforward. Even people that didn’t know what NuGet was were able to setup feeds and upload their games in a matter of a minutes. That was really easy and I would encourage people to try it out. You guys are doing some awesome stuff!

One thing I’ve noticed with NuGet that tripped people up is versioning, sometimes the version numbers weren’t sorted in the way we’ve expected but we’ve tackled that. I believe Xavier had a blog post about this.

Another thing is most people seem to be focused on creating or consuming packages and that is also what all the documentation on the NuGet site is about. Once you realize that NuGet can also be used as a distribution system of packages for your own application, things get more interesting. You can get started developing such thing by adding the NuGet.Core assembly to your project and navigating around with IntelliSense. The whole system is pretty slick and I’m using it in other applications I work on as well.

NuGet has a lot more uses than what can be seen at first sight, there is a whole lot to explore!”


Thank you Kelly for this chat, really appreciated!

Just like OCTGN, if you are working on an open source project you can get an Open Source subscription plan for your project!

Happy packaging!

MyGet is running over HTTPS only

A couple of weeks ago we've informed you that from July 1st, 2013, MyGet would be switching to HTTPS traffic only. We have just deployed these changes and from now on, your feed URL should be prefixed with https:// instead of http://.

Haven't made the switch yet? No problem: we will be redirecting most traffic to the new HTTPS endpoint. However this may add a little latency to your requests. Hence it's best to switch your URL now if you haven't done so.

To help in updating feed URLs on developer machines, you can make use of package source discovery. http://docs.myget.org/docs/reference/package-source-discovery
In short, every developer can issue the following commands in his/her Visual Studio Package Manager Console to update feed URLs:

Install-Package DiscoverPackageSources
Discover-PackageSources -Url https://www.myget.org/Discovery/Feed/ -OverwriteExisting

Protecting the security and privacy of our users is one of our most important tasks at MyGet. The fact that you can safely store your intellectual property on our servers is the best proof of that. We are confident this one-time change will make the entire MyGet experience even more secure.

Do not hesitate to contact us if you have additional questions through the comments below.

Happy packaging!

PS: If you are a MyGet Enterprise customer and are using your own domain name, no action isrequired on your side.

Switching to full HTTPS on July 1st, 2013

Important: a change is coming to URLs of MyGet. Please read through this post carefully as there may be some actions required on your side.

Protecting the security and privacy of our users is one of our most important tasks at MyGet. The fact that you can safely store your intellectual property on our servers is the best proof of that.

Currently, MyGet supports both http as well as https to communicate with our applications. To further improve our security, we're removing http access in the near future and will be switching to https only by July 1st, 2013, using a 2048-bit key certificate. By using only https, we can guarantee a secure communication channel between you and our servers.

Unfortunately, this change may require some action on your side. We will be discontinuing the http://www.myget.org URL in favor of https://www.myget.org. This means:

  • Your developers and/or clients may have to update their configuration
  • Your continuous integration servers may have to be reconfigured to make use of this new URL

This transition will happen in the following stages:

Actions required on your end:

  • Before July 1st - All links to MyGet have to be migrated to the https://www.myget.org URL if you are not on the Enterprise plan.

To help in updating feed URLs on developer machines, you can make use of package source discovery. http://docs.myget.org/docs/reference/package-source-discovery
In short, every developer can issue the following commands in his/her Visual Studio Package Manager Console to update feed URLs:

Install-Package DiscoverPackageSources
Discover-PackageSources -Url https://www.myget.org/Discovery/Feed/


We are confident this one-time change will make the entire MyGet experience even more secure.

Do not hesitate to contact us if you have additional questions.

Best regards,
the MyGet team

A Glimpse into our toolbox

Every now and then, we like to give you some insight in our development and the tools we use. This time, let’s have a look at Glimpse. Glimpse gathers and presents detailed diagnostic information about the behavior and execution of your web application. It’s like Firebug, but for the server.

Glimpse can be installed by installing the Glimpse.Mvc4 package. Different packages exist for different frameworks. Once installed, we can navigate to the /glimpse.axd file to enable/disable Glimpse on our development machine. The links on this page are also bookmarklets which can be used to turn on/off Glimpse. Once enabled, here’s what we see: a nice overview of important timings for our current page.

Glimpse toolbar

We are using Glimpse on our development machines to get some simple diagnostics at a glance. And the fun fact of the day: Glimpse uses MyGet to publish nightlies. Interested in what’s going on with that project? Add their nightlies feed to Visual Studio through the Package Manager Console:

Install-Package DiscoverPackageSources
Discover-PackageSources -Url "https://www.myget.org/gallery/glimpsenightly"

 

From then on, you can add Glimpse from their nightlies feed. We’ve used these nightlies in the past weeks and discovered their new HUD (head-up-display) feature in there as well as a new look-and-feel for the Glimpse client.

We can click the Glimpse toolbar and explore our request. For example, we can fetch a list of all actions and attributes that are being executed in ASP.NET MVC:

ASP.NET MVC Pipeline

A complete timeline of our requests is available as well:

Glimpse request timeline

We are also able to intercept and debug AJAX requests. If you are using Entity Framework or ADO.NET, expect to see your queries in here. If you’re developing mobile web applications, expect to be able to intercept remote calls as well. And the best thing: this is open source!

Happy packaging! And happy Glimpsing!

Build services: support for build.cmd and build.ps1

Using MyGet Build Services, you have the opportunity to control exactly how your project gets built. MyGet Build Services will scan the contents of your source control and look for a file it can work with.

In short, the following files are searched for (in order of precedence):

  • build.bat, build.cmd or build.ps1
  • MyGet.sln
  • Any other *.sln file
  • *.csproj or *.vbproj
  • *.nuspec

Build.bat and build.cmd can be simple batch files which perform builds and packaging. Build.ps1 can be a PowerShell script which will be invoked. Our Build Services overview blog post provides more detail on environment variables and tools that can be used.

Happy packaging!

New features in MyGet 1.7

We’re happy to announce we’ve completed another sprint. The main focus for this sprint was to start a redesign of our user experience. Next to that, new features have been introduced as well. Let’s have a look at what has changed and which cheese we moved.

A complete change log can be found on our new documentation site.

First steps in redesigning the MyGet experience

One of the first things you will notice when logging in to MyGet is that we’ve drastically changed the look and feel of the homepage. First of all, we decided the header we had earlier was too high and didn’t add much value. We’ve now condensed the header when authenticated. Your gravatar image will be shown and when hovering your username, a list of all feeds you have access to will be shown.

MyGet new design

The initial view you get is an activity stream. This provides the latest information about your feeds as well as the packages on it. On the right side, we’ve added quick navigation to all your feeds.

The feed details page now features a couple of additional buttons: you can clone a feed as well as delete a feed from that page.

Cheese has moved

We’re planning on further improvements in our next sprint!

New features and improvements

The following new features have been deployed:

In the coming days, we will be blogging about these features in more detail.

The page load speed of MyGet has improved as well. We’ve been working on optimizing file sizes, compression and are using CSS sprites for many of our images.

We hope you like this new drop. Let us know your thoughts in the comments below!

Happy packaging!

Create a list of favorite ReSharper plugins

With the latest version of the ReSharper 8 EAP, JetBrains shipped an extension manager for plugins, annotations and settings. Where it previously was a hassle and a suboptimal experience to install plugins into ReSharper, it’s really easy to do now. And what is really nice is that this extension manager is built on top of NuGet! Which means we can do all sorts of tricks…

The first thing that comes to mind is creating a personal NuGet feed containing just those plugins that are of interest to me. And where better to create such feed than MyGet? Create a new feed, navigate to the Package Sources pane and add a new package source. There’s a preset available for using the ReSharper extension gallery!

image_thumb[1]

After adding the ReSharper extension gallery as a package source, we can start adding our favorite plugins, annotations and extensions to our own feed.

image_thumb[3]

Of course there are some other things we can do as well:

  • “Proxy” the plugins from the ReSharper extension gallery and post your project/team/organization specific plugins, annotations and settings to your private feed. Check this post for more information.
  • Push prerelease versions of your own plugins, annotations and settings to a MyGet feed. Once stable, push them “upstream” to the ReSharper extension gallery.

Happy packaging!

Improved search syntax in NuGet client... and on your MyGet feeds

Yesterday, the NuGet team released some improvements to searching the official NuGet package source. Today, you can also use this new syntax on your MyGet feeds!

This new search syntax allows us to narrow our search to a particular attribute of a NuGet package. For example, we want to search for packages which contain “Glimpse” in the Id, we can type “id:glimpse”. We can also search the description of a package and check if it contains any of the given words. For example “description:twitter bootstrap” will yield packages containing either the word “twitter” or “bootstrap” in their description.

NuGet improved search syntax

Some example searches: (taken from the NuGet blog)

Attribute

Example

id

id:jQuery

title

title:Validation

description

description:dependency injection

authors

authors:Outercurve Foundation

tags

tags:silverlight

 

Enjoy, and as always…

Happy packaging!

Build services - an overview

Our 1.6 release shipped a number of interesting new features and enhancements to existing features, including those for MyGet Build Services. In this post, we’ll describe existing and new features and enhancements in a bit more detail.

One important thing to know is that Build Services is intended to make packaging projects easier. We are not aiming to become a full CI server like TeamCity of Team Foundation Server. That said, we do believe Build Services is appropriate for many scenarios and can take a lot of work out of your hands. We found some blog posts you may like: Nico Vermeir is using Build Services to package Windows Phone libraries. We also use it ourselves to push components from GitHub to NuGet.org.

Specify build configuration and platform

When creating or modifying a build source, you can now specify the configuration and platform that should be built. Easily switch between Release (the default configuration) or Debug or any other configuration that exists in your solution. The same goes for platform: if you wish to specifically build for x86, this can now be specified as the target platform.

Build configuration and platform

Note that we also set a %Configuration% and %Platform% environment variable during the build process. This way you can make use of these in your customized builds that are run using a build.bat file.

Package versioning

In earlier versions of MyGet Build Services, we’ve been generating a random incremental version number for packages created during build. From now on, we have support for true incremental build numbers through the build source settings

Specify package version number

The build counter starts with zero and increments with 1 on every build. You can also specify a version format ( Iuse '{0}' as a placeholder for the build counter) which will be generated during build. Note that if you enable the forbid packages which are non-compliant with Semantic Version option ,you should make sure the version format specified follows the Semantic Version rules.

We also set a %BuildCounter% and %PackageVersion% environment variable during the build process. This way you can make use of these in your customized builds that are run using a build.bat file.

Supported project types

We’ve added support for psake builds (use a built.bat file and invoke psake). Building Windows Phone 8 packages is now also supported out of the box. This brings us to the following list of supported frameworks and SDK’s:

  • .NET 2.0, .NET 3.0, .NET 3.5, .NET 4.0 and ,NET 4.5
  • PCL support (2012)
  • Windows Azure SDK
  • Windows Identity Foundation SDK
  • Silverlight 4, Silverlight 5
  • TypeScript SDK
  • psake

Regarding test runners, we now support

  • MsTest
  • MbUnit 2, MbUnit 3
  • NBehave
  • NUnit (up to 2.6.2)
  • xUnit.net
  • csUnit
  • RSpec

We believe adding these SDK’s out-of-the-box provides a lot of value to our users and we want to continue investing in expanding the number of supported SDK’s.

Environment variables

As part of the build process, we now set the following environment variables. Note that these are all read-only.

%BuildRunner%

Always "MyGet", can be used to determine if running on MyGet Build Services

%NuGet%

Path to a maintained-by-MyGet NuGet.exe

%SourcesPath%

Path to source code being built

%Configuration%

Build configuration (defaults to Debug)

%Platform%

Platform to build (defaults to blank)

%VersionFormat%

Version format specified in build configuration

%BuildCounter%

Build counter value

%PackageVersion%

%VersionFormat% with %BuildCounter% filled in, used as the auto-generated package version number

%EnableNuGetPackageRestore%

NuGet package restore enabled? Always true.

Other features

Some smaller features have been implemented for version 1.6 as well.

  • When building from GitHub/BitBucket, a link to the changeset that has been built is available from the build services tab.

image

  • Hanging build detection: whenever a build hangs for > 15 minutes, we kill the build process and set the build status to failed.
  • The build log can be copied to clipboard.
  • Build status is refreshed automatically.

Happy packaging!