MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Retention Rules and Package Pinning

When using MyGet feeds for package archiving or nightly builds, a lot of packages show up on the feed very quickly. By default, we keep all package versions available on your feed. If you would like to do some automated housekeeping, retention rules can be added per feed. Whenever a package is added to your feed, we'll make sure these retention rules are respected.

Package retention rules for a feed

Our latest release added some additional options for package retention. You can now:

  • Specify the number of stable versions to keep
  • Specify the number of prerelease versions to keep
  • Specify if we have to keep depended packages or not

That last one is different from what we did in the past. We used to blindly delete packages that were older than version X, even if there were packages depending on it. From now one, we will keep packages that are depended on unless specified through the settings.

Pinning Package Versions

Certain packages should never be deleted automatically. For every package, we now support pinning specific versions – a pinned package will never be deleted by the retention rules. From the package details, this can be done using the pin/unpin buttons.

Pinning packages - ignore retention rules

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Release notes for MyGet 1.8

MyGet 1.8 was released on September 10, 2013. We will blog about new features in the next days and weeks.

Features

MyGet

  • Support for NuGet 2.7
  • Metadata for packages is auto-updated from upstream feeds
  • Retention policies: pin packages so they don't get deleted
  • Retention policies: packages that are depended on will no longer be deleted (unless explicitly enabled)
  • Push upstream: package source code repositories can be labeled when pushing packages upstream
  • Send e-mail when feed permissions change
  • Users can revoke their own access from a feed
  • Automatic mirroring for packages from upstream feeds when feed proxy is enabled

MyGet Enterprise

  • Administrators can now join a feed. Feed owners are notified of this action.

MyGet Build Services

  • Repositories from GitHub organizations are now shown
  • The latest build version is shown in the UI
  • Package sources added at the feed level are available on the build server
  • Automatic support for NuGet package restore even if it's not enabled for the solution
  • Support for NuGet 2.7 package restore - see http://docs.myget.org/docs/reference/build-services#Package_Restore
  • Package sources added at the feed level are available on the build server for package restore
  • Build labeling: on succesful or failed builds, a label can be added to the sources. This is compatible with GitHub Releases. - see http://docs.myget.org/docs/reference/build-services#Sourcelabeling(tagging)
  • Support for MyGet.cmd, MyGet.bat, MyGet.ps1

Bug Fixes

  • Fixed an issue with copy to clipboard
  • Fixed an issue when pushing packages upstream to authenticated feeds
  • Support page is no longer behind an authentication wall
  • Fixed an issue when build services used @ in the username
  • Various performance improvements

Happy packaging!

Downtime September 11 and 12, 2013 - Root cause

On September 11 and 12, 2013, we have experienced some downtime. The website and back-end services have crashed a number of times across all instances, bringing our system to a halt a couple of times. We hate downtime as much as you do and want to apologize for the trouble this may have caused you. Let’s have a look at the symptoms and root cause.

Symptoms

Twitter and e-mail notified us of this a good 2 minutes before our monitoring kicked in. When we had a look at our status dashboards, we saw high CPU usage on all machines, inexplicable crashes of machines and an error message in the Windows event logs right before the machines died and restarted.

In our back-end queue, where all work is added and waits to be processed, we found 25 messages that seemed to be retried over and over again, having been dequeued over 150 times.

We have manually removed these messages from the queue and inserted them again after the website was running stable again. After monitoring the situation for a couple of hours, all systems seemed to be running stable again. Seemed…

A number of hours later, we received some additional monitoring events. The problem was back! This time we decided not to fire fight but dig deeper into the issue we were facing. We took the site offline and connected to the machines to analyze event logs and so on. Except for the error message in the Windows event log, nothing was going on. Next, hooked a debugger to it, waiting for anything to happen.

The issue

Perhaps a root cause analysis is not the best moment to blog about our new features and updates in our latest deployment, but we have to in this case. Last Tuesday, we deployed our 1.8 release which brings an update to retention policies. These allow MyGet to automatically delete packages from a feed after a certain number of packages has been added, an automatic feed cleanup as it were.

The updated retention policies feature now tries not to break a dependency chain. Before, packages that were depended on by other packages but were subject to the retention policy settings would be deleted. Now, the retention policy handler respects the entire dependency chain for a package and will no longer remove depended packages. This, of course, requires us to parse the entire dependency tree for a package.

imageNow what happens if package A depends on package B, B depends on C and C depends on B? Right: that’s a potential infinite loop. Guess what was happening: a StackOverflowException, crashing our entire machine without giving it the chance to write output to our error logs.

Because of the crash, the message in our backend was requeued over and over. We typically move a message to a dead letter queue after a number of retries (typically 32). However because of the crash, that logic couldn’t kick in and move the message out, resulting in the message being requeued over and over again, triggering the same behavior again.

While we test our application thoroughly before going to production, this was an edge case. Who would create a package with circular dependencies anyway? We found that on one feed, a circular package dependency did exist.

Solution

We have made a fix to our algorithm which now supports this scenario and will stop following the circular dependency.

After confirming our algorithm, we have deployed it at 6 AM (CET) this morning, resolving the issue. The messages that were stuck in our queue (and triggered the initial issue) have been requeued to confirm correct working and  were all processed successfully.

Our status page can always be found at http://status.myget.org, showing uptime of our last months as well as the outage of the past hours.

Again, we do apologize for the inconvenience caused.

Happy packaging!