Twitter University
Twitter University
  • Видео 95
  • Просмотров 1 812 600
How We Learned to Stop Worrying and Love Fan-In at Twitter
Twitter's massive write-time and nearly-instantaneous Tweet fanout system is well-known. This talk will discuss the architecture and challenges involved in the shift towards read-time determination and collection of Tweets for users, incorporating disparate content relevance and recommendation systems. With additional layers of software business logic composed into the core Twitter experience, this has necessitated the adoption of new contracts between teams and services, a re-examination of our principles about resiliency, and reconciliation between competing forms of content selection.
Speaker: Mike Cvet (@mikecvet)
For more info, see qconsf.com/sf2016/presentation/how-we-learned-stop-wor...
Просмотров: 13 307

Видео

Scaling Reliability at Twitter: So You Want to Add a 9
Просмотров 2,5 тыс.7 лет назад
We choose microservice architectures for many different reasons, including, often, improving reliability. However, there is a dark side. Modular systems have many benefits, but as you add more components, each of which can fail, it makes the overall system more fragile, because there are new ways in which it can fail that previously didn't exist. From its inception, Finagle was designed as a to...
Building Twitter’s Next-Gen Alerting System
Просмотров 2,8 тыс.7 лет назад
Twitter’s Observability team provides Twitter developers a monitoring infrastructure including real time dashboards and alerting for their services. Twitter has grown orders of magnitude since it debuted at SXSW. The monitoring infrastructure has followed suit, ingesting orders of magnitude more metrics, from millions to billions. As the company grew, so did the challenges of providing an alway...
JVMs across the Data Center at Twitter
Просмотров 1,3 тыс.7 лет назад
Twitter services have to be able to handle the flood of requests, responses and actions required to process hundreds of millions of tweets every day. This talk will cover how we schedule and manage services in our data centers to ensure reliability and performance, as well as the challenges (and some solutions!)  encountered in getting the JVM and JDK to run our Scala- and Java-based services t...
Managing Big Storage Clusters at Twitter
Просмотров 1,6 тыс.7 лет назад
Distributed databases are complex systems with multiple dependencies that must be maintained in order to conserve correctness. Manhattan, Twitter's primary distributed key-valve store, has many differing capabilities that require careful and subtly different orchestration while operating on the cluster in order to deploy new versions or add and remove nodes. This talk will touch on the evolutio...
Hardware & Provisioning Engineering at Twitter
Просмотров 1,7 тыс.7 лет назад
Twitter operates hundreds of thousands of servers. At that scale, repairs and reallocations are frequent and each requires a re-installation of the system. The Provisioning Engineering system at Twitter, called Wilson, is designed to handle every part of a server's lifecycle that does not require a person with a screwdriver. From bios updates to RAID controller configuration and OS installation...
Closing remarks from Chris Pinkham, VP of Platform Engineering
Просмотров 9938 лет назад
Closing remarks from Chris Pinkham, VP of Platform Engineering
AWS + Apache Aurora/Mesos at Twitter
Просмотров 2 тыс.8 лет назад
In 2014, TellApart (now part of Twitter) adopted Apache Mesos and Apache Aurora as the solution to run their infrastructure in AWS. Engineering Manager David Hagar shares the team’s experiences, the problems encountered, and how they addressed them. He also gives a glimpse into the work to extend the capabilities of the Compute infrastructure across private and public clouds.
Twitter Heron on Apache Aurora
Просмотров 2,4 тыс.8 лет назад
Heron is the next generation real-time streaming analytics platform used heavily in Twitter. Heron runs diverse real-time analytics applications ranging from counting to real-time machine learning. Heron tech lead Maosong Fu talks about how Heron leverages Apache Aurora extensively to run its topologies.
Aurora Workflows: Deploying on Apache Aurora at Twitter
Просмотров 2,9 тыс.8 лет назад
One of the biggest cognitive overheads for a developer is to shepherd a deployment from development to production. Software engineer David Mclaughlin from the Compute team talks about our internal project called “Workflows” which aims to reduce this overhead and become a key building block for continuous deployment automation in the future.
Chargeback for multi-tenant Infrastructure systems
Просмотров 1,9 тыс.8 лет назад
Software engineers on Cloud Infrastructure Management, Vinu Charanya and Jessica Yuen, talk about how they built a generic system that helps define chargeable resources, collects utilization metrics, resolves owners, and generates billing and utilization reports to improve resource utilization and cost effectiveness of every multi-tenant infrastructure service at Twitter.
Managing a large scale compute platform (Apache Aurora/Mesos)
Просмотров 2,7 тыс.8 лет назад
David Robinson, site reliability engineer on Compute, talks about how they leverage Audubon, Wilson, and other tooling to manage (configuration, deployment, and other operational procedures) across tens of thousands of hosts in the Compute cluster (powered by Apache Aurora/Apache Mesos). He also shares the challenges when operating at this scale and things that can potentially go wrong that imp...
Server Provisioning and Management at Scale
Просмотров 7 тыс.8 лет назад
Tech lead of provisioning engineering, Eric Danielson talks about two specific systems: Audubon - Machine database and Wilson API/Lifecycle Manager - Machine Lifecycle Manager that are used to provision, track, and manage a fleet of tens of thousands of hosts at Twitter.
Live Code Updates
Просмотров 2,9 тыс.9 лет назад
Update running Java code without restarting your VM!
Apache Aurora Adopters Panel
Просмотров 8529 лет назад
Apache Aurora Adopters Panel
Operating Apache Aurora and Mesos at Twitter
Просмотров 18 тыс.9 лет назад
Operating Apache Aurora and Mesos at Twitter
Apache Aurora and Mesos at TellApart
Просмотров 3,5 тыс.9 лет назад
Apache Aurora and Mesos at TellApart
Building First Class SDKs: A Fabric Case Study
Просмотров 1,1 тыс.9 лет назад
Building First Class SDKs: A Fabric Case Study
Building Fabric for Android Apps
Просмотров 24 тыс.9 лет назад
Building Fabric for Android Apps
Paradox of the Cloud
Просмотров 3,2 тыс.9 лет назад
Paradox of the Cloud
The Binary Compatibility Challenge
Просмотров 4,7 тыс.9 лет назад
The Binary Compatibility Challenge
What Ferris Bueller Taught Me About Interaction Design
Просмотров 14 тыс.9 лет назад
What Ferris Bueller Taught Me About Interaction Design
Cascading: A Java Developer's Companion to the Hadoop World
Просмотров 7 тыс.9 лет назад
Cascading: A Java Developer's Companion to the Hadoop World
Running Mesos at Atlassian
Просмотров 3,9 тыс.9 лет назад
Running Mesos at Atlassian
Continuous Deployment with Singularity
Просмотров 2,7 тыс.9 лет назад
Continuous Deployment with Singularity
How Airbnb Simplifies with Mesos
Просмотров 6 тыс.9 лет назад
How Airbnb Simplifies with Mesos
Reactive Stream Processing Using Apache Mesos
Просмотров 3 тыс.9 лет назад
Reactive Stream Processing Using Apache Mesos
Simplifying with Mesos and Marathon
Просмотров 17 тыс.9 лет назад
Simplifying with Mesos and Marathon
Spark on Mesos
Просмотров 5 тыс.9 лет назад
Spark on Mesos
Highly Distributed DevOps
Просмотров 3,2 тыс.9 лет назад
Highly Distributed DevOps

Комментарии

  • @TristanBailey
    @TristanBailey 5 месяцев назад

    Great talk and learned things on the side of main topic

  • @AbdulRauf-yy2ws
    @AbdulRauf-yy2ws 5 месяцев назад

    Indeed, I have learned a lot today. Thanks Doug.

  • @alihammadshah
    @alihammadshah 5 месяцев назад

    I was looking for this, watch it first when docker launched. Docker had never been a job requirement but it is now, the first resource I could remember to learn from was this.

  • @2girlstyrick
    @2girlstyrick 7 месяцев назад

    Elon is rebooting the vine app

  • @mastanraovisarapu5501
    @mastanraovisarapu5501 Год назад

    What is the maximum repetition level for parquet?

  • @JerryCheevers
    @JerryCheevers Год назад

    Vijaya Gadde needs to be in fucking jail for LIFE!

  • @maryjeanestacio5257
    @maryjeanestacio5257 Год назад

    😊😊😊

  • @PentaRaus
    @PentaRaus Год назад

    Turns out even women of color can be toxic, even treasonous.

  • @frankwren8215
    @frankwren8215 Год назад

    If I were you I wouldn't leave videos like this up, Gadde in particular is poison to businesses especially after her now well known illegal actions.

  • @mortenbjoernsvik6550
    @mortenbjoernsvik6550 Год назад

    I build pex files on windows and inject them into minimal containers running alpine linux. As long as the base glibc dev libraries are present it works. Would love to take it a step further, one pex with all dependencies enclosed.

  • @mrgamerguy321
    @mrgamerguy321 Год назад

    All these people and no security…. 🤷‍♂️ No one is brave enough to step up and make the proper changes …. No security because the feel emboldened and untouchable… We should show them just how touchable they are….

  • @imrael496
    @imrael496 Год назад

    Vijaya discredits these women. She never embraced the constitution. She thought dhe could crush free speech.

  • @Youtuber-xs9cp
    @Youtuber-xs9cp Год назад

    This is like Miss universe contest or something. This is what woke twitter was wasting time on money on. No wonder they were performing horribly. Those were perverted Parag Agwaral was focusing on. Soon he will become Harvey Weinstein roommate.

  • @joelm6631
    @joelm6631 Год назад

    Thanks Elon

  • @bshak
    @bshak Год назад

    Amazing insight into the chargeback concepts and its implementation in such a complex system. Great job!

  • @turolretar
    @turolretar Год назад

    But like um and yeah uhh

  • @cdes68
    @cdes68 2 года назад

    You sold out pisserZ.

  • @Krazydurnut
    @Krazydurnut 2 года назад

    I love twitter

  • @kennethkath6527
    @kennethkath6527 2 года назад

    Informative

  • @num137
    @num137 2 года назад

    This guy invented EC2, i.e. cloud virtual servers, i.e. the core of the cloud on AWS. Basically one of the inventors of the cloud.

  • @astrobot4621
    @astrobot4621 2 года назад

    My account got locked but IM NOT BREAK RULES

  • @RR2Ch
    @RR2Ch 2 года назад

    RIP TWITTER 2006-2022:(

  • @sinistar3198
    @sinistar3198 2 года назад

    Is there a link to the project mentioned around 32:00 ?

  • @franklinclinton7761
    @franklinclinton7761 2 года назад

    Twitter, would u stop banning twitter users for life. I can understand that Tweets that violate the rules should get taken down like calling for violence and other stuff. But some times users are suspended when they can’t verify their account because they lost their access to their email address and twitter code. Or when User insult other users after they were insulted by other users. One could just take down the Tweets or ban the users temporarily for a few hours or days but not permanent ban for ever. There is no freedom of speech on your platform. Become more democratic

    • @ryeguy01
      @ryeguy01 2 года назад

      this is a tech talk not a support forum

  • @edwrdtvso
    @edwrdtvso 2 года назад

    #Hyve Solutions ZEUS 1U RACK

  • @jg-jsbg4725
    @jg-jsbg4725 2 года назад

    HOW WE LEARNED TO STOP WORRYING AND LOVE FAN-IN AT TWITTER

  • @jg-jsbg4725
    @jg-jsbg4725 2 года назад

    HEY ☀️☪️️ TWITTER UNIVERSITY

  • @googlegoogle9712
    @googlegoogle9712 2 года назад

    All are fukable except that last one on far right.

  • @jgonzalesm6
    @jgonzalesm6 2 года назад

    Bye bye people. Elon Musk has taken over to allow free speech....something the "old guard" censored to include the Hunter Biden laptop issue.

  • @glockdude5472
    @glockdude5472 2 года назад

    I can’t wait until all these woke people get canned by Musk. Karma sucks

  • @BlastinRope
    @BlastinRope 2 года назад

    We all agree, mistakes were made

  • @davidpina9605
    @davidpina9605 2 года назад

    All to bad the censorship campaign ended for a sec but you know it will be back with full force those plebs are so dumb and uninformed. Keep the tears coming Vijaya Gadde seems like that's how you get everything in life you want. As above so below... Nov 23 2023 🎆

  • @fredsavage7380
    @fredsavage7380 2 года назад

    Smells like fish up there

  • @merrymeagain6366
    @merrymeagain6366 2 года назад

    Oprah Winfrey the last Queen⛳.........🖤 🌰🐒💭

  • @merrymeagain6366
    @merrymeagain6366 2 года назад

    Atari Nintendo Super Nintendo Playstation=40years of computer love gone top gun joy boy stick.......stay with me goose......🗑list waste world-wide made in China day1 management.......🚬 Help me-Help YOU winfree........🖤queen👣

  • @grimonce
    @grimonce 2 года назад

    Why do I only find out about it in 2022....? I see pants can generate pex files as well? It wasn't made quite clear in the video, but do PEX archive contain a full interpreter with all the dependencies? Like cython or pyinstaller can do? Can we use these to distribute software in a single archive to users that don't have Python installed?

    • @MatthewMartinDean
      @MatthewMartinDean Год назад

      For future readers, the answer is no, pex does not include an installed python interpreter and expects one to already be on the target machine.

    • @grothaal
      @grothaal 10 месяцев назад

      @@MatthewMartinDean does it carry all dependencies though? Like, can I use it to package my application in dev and move it to a production server?

  • @leoxiaoyanqu
    @leoxiaoyanqu 2 года назад

    Only 51K views? Definitely a "hidden" gem for anyone (like me) working in search! Thanks for sharing the talk! Wish I'd be sitting there in the audience some day.

  • @PWNINSWAGMASTER
    @PWNINSWAGMASTER 3 года назад

    TELL DORSEY TO STOP TARGETING CONSERVATIVES like the freakin bully he is!!!! He just attacked Rubin even though he cited LEFT-wing sources!

  • @AnRodz
    @AnRodz 3 года назад

    why is your shell not case sensitive? @2:41

  • @nicholascrews744
    @nicholascrews744 3 года назад

    @ ruclips.net/video/Q5POuMHxW-0/видео.html It's funny how every engineer's first impulse, out of literally an infinite number of ways to modify the filesystem, is to `rm -rf /`. Like, other options exist, man!

  • @PeopleBeStrong
    @PeopleBeStrong 3 года назад

    Old but good

  • @PeopleBeStrong
    @PeopleBeStrong 3 года назад

    Nice

  • @PeopleBeStrong
    @PeopleBeStrong 3 года назад

    Good Video

  • @julabernier3915
    @julabernier3915 3 года назад

    CUI CUI

  • @J1GS4W0
    @J1GS4W0 3 года назад

    How has changed docker from 7years to today? Is this video deprecated?

  • @erikhiga
    @erikhiga 3 года назад

    The CEO of Sony who is my friend is flying to the USA to appear in court on my behalf against you.

  • @samzofkie8779
    @samzofkie8779 3 года назад

    str8 2 da killer app

  • @ButtBanginTranny
    @ButtBanginTranny 3 года назад

    Watch one of the witches sweat and panic on Joe Rogan podcast 🤣🤣

  • @divakaryadav8174
    @divakaryadav8174 3 года назад

    your voice is 100% same as the Elon Musk.

  • @dori8118
    @dori8118 3 года назад

    THAT WAS ON 2013,,,,,,,,,,,,,,,,,,,,